Oct 29 22:11:28.080869 ip-10-0-128-31 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 29 22:11:28.080899 ip-10-0-128-31 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 29 22:11:28.080907 ip-10-0-128-31 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 29 22:11:28.081233 ip-10-0-128-31 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 29 22:11:38.145920 ip-10-0-128-31 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 29 22:11:38.145933 ip-10-0-128-31 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ae1e5626b3c245d19718833302044535 --
Oct 29 22:13:36.490591 ip-10-0-128-31 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:13:36.862170 ip-10-0-128-31 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:13:36.862170 ip-10-0-128-31 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:13:36.862170 ip-10-0-128-31 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:13:36.862170 ip-10-0-128-31 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:13:36.866650 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866575    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:36.866650 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866646    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:36.866650 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866650    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:36.866650 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866654    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:36.866650 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866656    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866659    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866663    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866666    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866668    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866671    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866675    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866677    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866680    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866683    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866686    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866688    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866691    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866693    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866696    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866698    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866707    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866709    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866712    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866715    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:36.866835 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866718    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866720    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866723    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866725    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866728    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866731    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866733    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866736    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866738    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866741    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866744    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866747    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866749    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866752    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866755    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866758    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866760    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866763    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866765    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:36.867290 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866768    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866771    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866773    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866776    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866779    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866781    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866783    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866787    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866790    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866793    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866795    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866798    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866800    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866803    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866806    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866810    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866813    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866816    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866819    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866821    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:36.867750 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866824    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866826    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866829    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866832    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866834    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866837    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866840    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866843    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866845    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866848    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866851    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866853    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866856    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866858    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866861    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866863    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866866    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866868    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866871    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:36.868254 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866873    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866878    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866883    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866886    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866890    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866894    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866897    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866900    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866903    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866906    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866909    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866912    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866914    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866917    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866919    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866922    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866925    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866927    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866930    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:36.868710 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866933    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866936    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866940    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866946    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.866950    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867702    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867709    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867713    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867716    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867718    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867721    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867724    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867726    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867729    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867732    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867735    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867738    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867741    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867744    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:36.869196 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867746    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867749    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867768    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867772    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867776    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867779    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867783    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867787    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867791    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867795    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867798    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867801    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867803    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867807    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867810    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867812    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867815    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867817    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867820    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867822    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:36.869668 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867825    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867827    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867830    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867832    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867835    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867838    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867840    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867843    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867846    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867850    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867853    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867856    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867859    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867862    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867865    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867868    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867870    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867874    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867877    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:36.870146 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867879    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867882    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867884    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867887    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867890    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867892    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867895    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867898    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867900    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867903    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867906    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867908    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867911    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867913    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867916    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867918    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867921    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867923    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867926    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867929    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:36.870604 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867931    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867934    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867936    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867939    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867942    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867945    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867951    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867953    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867956    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867958    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867961    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867964    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867966    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867969    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867971    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867974    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867976    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867979    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867982    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867985    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:36.871121 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867987    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867990    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867993    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867995    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.867998    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868000    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868003    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868006    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868009    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868012    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868014    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868017    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868019    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868506    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868511    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868515    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868518    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868520    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868525    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:36.871602 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868528    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868531    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868534    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868536    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868539    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868542    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868544    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868547    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868550    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868552    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868555    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868558    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868561    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868563    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868566    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868569    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868571    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868574    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868576    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868579    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:36.872068 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868581    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868584    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868587    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868589    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868592    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868595    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868597    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868600    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868602    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868605    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868608    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868612    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868614    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868617    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868619    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868622    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868625    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868627    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868631    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:36.872556 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868634    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868637    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868640    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868642    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868646    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868649    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868651    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868654    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868656    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868659    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868661    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868664    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868667    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868669    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868672    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868675    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868678    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868680    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868683    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:36.873128 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868686    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868688    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868691    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868693    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868696    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868700    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868702    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868705    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868707    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868710    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868713    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868715    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868718    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868721    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868724    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868726    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868729    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868732    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868734    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:36.873583 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868737    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868741    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868745    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868748    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868751    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868753    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868756    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868758    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868761    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868763    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868766    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868768    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868771    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868773    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868776    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868779    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868782    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868784    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868789    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:36.874028 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868791    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868794    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868796    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.868799    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868886    2583 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868894    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868899    2583 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868904    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868908    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868911    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868916    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868928    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868932    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868935    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868938    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868941    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868944    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868947    2583 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868950    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868953    2583 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868956    2583 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868959    2583 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868962    2583 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868965    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:13:36.874479 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868968    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868971    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868974    2583 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868978    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868981    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868984    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868988    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868992    2583 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868996    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.868999    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869002    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869005    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869009    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869013    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869018    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869022    2583 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869027    2583 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869031    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869038    2583 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869044    2583 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869048    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869051    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869054    2583 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869059    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:13:36.875068 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869062    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869065    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869068    2583 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869071    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869074    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869077    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869080    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869083    2583 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869086    2583 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869089    2583 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869093    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869096    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869099    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869103    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869106    2583 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869110    2583 flags.go:64] FLAG: --help="false"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869114    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-128-31.ec2.internal"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869119    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869122    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869126    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869129    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869132    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869135    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:13:36.875691 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869138    2583 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869141    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869144    2583 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869147    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869150    2583 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869153    2583 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869156    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869159    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869162    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869165    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869168    2583 flags.go:64] FLAG: --lock-file=""
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869170    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869173    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869176    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869181    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869184    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869187    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869190    2583 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869193    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869197    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869200    2583 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869203    2583 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869207    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869210    2583 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869214    2583 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:13:36.876238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869217    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869220    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869224    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869227    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869230    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869233    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869236    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869243    2583 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869246    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869249    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869252    2583 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869255    2583 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869257    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869260    2583 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869263    2583 flags.go:64] FLAG: --port="10250"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869266    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869269    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a97299f08126dc40"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869272    2583 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869275    2583 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869278    2583 flags.go:64] FLAG: --register-node="true"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869281    2583 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869285    2583 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869287    2583 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869290    2583 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869293    2583 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869296    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:13:36.876859 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869300    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869304    2583 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869307    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869310    2583 flags.go:64] FLAG: --runonce="false"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869313    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869316    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869319    2583 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869332    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869335    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869339    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869342    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869346    2583 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869348    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869351    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869354    2583 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869357    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869360    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869363    2583 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869366    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869371    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869374    2583 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869377    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869381    2583 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869384    2583 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869387    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:13:36.877491 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869389    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869392    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869395    2583 flags.go:64] FLAG: --v="2"
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869400    2583 flags.go:64] FLAG: --version="false"
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869403    2583 flags.go:64] FLAG: --vmodule=""
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869407    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869410    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869536    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869542    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869545    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869548    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869551    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869554    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869556    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869559    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869562    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869564    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869568    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869570    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869573    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869575    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:36.878072 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869578    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869580    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869583    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869586    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869588    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869590    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869593    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869596    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869598    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869601    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869603    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869606    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869609    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869611    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869614    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869617    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869619    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869621    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869624    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869628    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:36.878590 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869631    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869633    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869636    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869638    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869641    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869643    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869646    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869648    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869651    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869654    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869657    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869659    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869662    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869664    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869667    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869669    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869672    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869674    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869678    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:36.879054 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869683    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869686    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869688    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869692    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869695    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869697    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869700    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869702    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869705    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869708    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869710    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869712    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869716    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869719    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869721    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869724    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869727    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869730    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869732    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869735    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:36.879517 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869737    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869740    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869742    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869745    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869748    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869751    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869753    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869756    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869758    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869761    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869764    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869766    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869769    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869771    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869774    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869777    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869781    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869785    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869788    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:36.880021 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869790    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869793    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869796    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869798    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869801    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869804    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869807    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869810    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869812    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869815    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869817    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869820    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869822    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869825    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.869850    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869885    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:36.880482 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869889    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869894    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869897    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869900    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869902    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869905    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869907    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869910    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869913    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869916    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869918    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869921    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869924    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869928    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869930    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869933    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869936    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869939    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869941    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869944    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:36.880864 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869946    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869950    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869954    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869958    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869961    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869964    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869967    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869969    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869972    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869974    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869977    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869979    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869982    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869985    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869988    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869990    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869993    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869995    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.869998    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:36.881448 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870001    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870003    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870006    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870008    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870011    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870013    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870018    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870022    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870025    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870027    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870030    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870033    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870035    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870038    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870042    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870045    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870047    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870050    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870052    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:36.881894 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870055    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870058    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870060    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870063    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870065    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870068    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870070    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870073    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870075    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870078    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870081    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870083    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870086    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870088    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870091    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870094    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870096    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870098    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870101    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870105    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:36.882363 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870107    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870110    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870113    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870115    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870118    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870121    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870123    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870127    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870129    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870132    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870135    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870137    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870140    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870144    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870148    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870152    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870155    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870158    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870161    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:36.882841 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870163    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870166    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870168    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870172    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870175    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870177    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870179    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.870182    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.876359    2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.876375    2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.876388    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.876393    2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876449    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876455    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876459    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876462    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876465    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876468    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876471    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876473    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:36.883277 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876476    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876478    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876481    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876484    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876486    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876489    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876492    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876494    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876497    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876499    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876502    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876505    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876507    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876510    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876512    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876515    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876517    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876520    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876523    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:36.883761 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876525    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876528    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876539    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876542    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876545    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876548    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876550    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876553    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876556    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876558    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876561    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876564    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876566    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876569    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876571    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876574    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876576    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876579    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876582    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:36.884193 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876584    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876587    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876589    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876592    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876594    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876597    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876600    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876602    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876605    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876607    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876610    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876613    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876615    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876618    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876621    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876624    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876632    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876635    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876638    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:36.884669 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876640    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876643    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876645    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876648    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876650    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876653    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876655    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876658    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876661    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876663    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876666    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876668    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876671    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876673    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876676    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876679    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876681    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876683    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876687    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:36.885160 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876691    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876694    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876697    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876700    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876704    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876707    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876709    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876712    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876715    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876719    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876723    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876731    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876734    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876737    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876739    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876742    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876745    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876747    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876750    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:36.885692 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876752    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876755    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876757    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.876784    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876965    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876969    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876972    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876975    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876978    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876980    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876983    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876986    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876988    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876991    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876993    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876996    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.876998    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:36.886141 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877001    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877004    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877006    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877009    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877012    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877014    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877017    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877019    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877029    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877032    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877035    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877037    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877040    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877042    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877045    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877048    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877050    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877053    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877055    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:36.886551 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877058    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877060    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877063    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877065    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877068    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877070    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877073    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877075    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877078    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877080    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877083    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877085    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877088    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877091    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877093    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877096    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877098    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877101    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877104    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877107    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:36.886978 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877109    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877112    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877121    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877125    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877128    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877131    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877134    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877136    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877139    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877141    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877143    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877146    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877148    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877151    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877153    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877156    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877158    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877160    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877163    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:36.887440 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877165    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877168    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877170    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877173    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877181    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877184    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877186    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877189    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877192    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877194    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877197    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877199    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877202    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877204    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877207    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877209    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877217    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877220    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877222    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877225    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:36.887875 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877227    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877230    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877232    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877235    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877237    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877240    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877242    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877244    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877247    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877249    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877252    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877254    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877257    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877259    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877263    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:36.888505 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.877286    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877338    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877344    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877347    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877350    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877353    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877356    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877359    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877361    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877365    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877369    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877372    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877376    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877378    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877388    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877390    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877393    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877395    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877398    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:13:36.889124 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877401    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877403    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877405    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877408    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877410    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877413    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877415    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877418    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877420    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877423    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877425    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877428    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877430    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877433    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877435    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877438    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877440    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877443    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877449    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:13:36.889814 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877451    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877454    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877457    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877459    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877462    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877464    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877467    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877469    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877472    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877480    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877483    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877485    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877488    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877490    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877494    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877497    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877499    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877502    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877505    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877507    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:13:36.890253 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877510    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877512    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877515    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877517    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877520    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877522    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877526    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877529    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877532    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877535    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877538    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877542    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877545    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877548    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877550    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877553    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877556    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877558    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877561    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:13:36.890754 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877563    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877566    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877568    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877578    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877580    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877583    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877585    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877588    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877590    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877593    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877595    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877598    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877600    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877603    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877605    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877608    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877610    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877612    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877615    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877617    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:13:36.891194 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877620    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877622    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877625    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877627    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877631    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877634    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877637    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877640    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877643    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:36.877645    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.877699    2583 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.880401    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.881212    2583 server.go:1014] "Starting client certificate rotation"
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.881313    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.881369    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.883365    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:13:36.891674 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.889748    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:13:36.898827 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.898810    2583 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:13:36.902111 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.902097    2583 log.go:25] "Validated CRI v1 image API"
Oct 29 22:13:36.903396 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.903373    2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:13:36.905853 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.905837    2583 fs.go:135] Filesystem UUIDs: map[46d5fa04-d99a-4652-b74c-cd747d20a41b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e8d78256-42ef-4420-8e89-2d24bb29ed06:/dev/nvme0n1p3]
Oct 29 22:13:36.905905 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.905853    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 29 22:13:36.910755 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.910656    2583 manager.go:217] Machine: {Timestamp:2026-10-29 22:13:36.909418468 +0000 UTC m=+0.308849728 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100098 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec277aa0c2c13a13ec9371816e96cc62 SystemUUID:ec277aa0-c2c1-3a13-ec93-71816e96cc62 BootID:ae1e5626-b3c2-45d1-9718-833302044535 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f9:b3:1c:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f9:b3:1c:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:a4:0c:6f:0f:7d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:13:36.910830 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.910756    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:13:36.910867 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.910852    2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:13:36.912640 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.912621    2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:13:36.912765 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.912641    2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:13:36.912809 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.912768    2583 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:13:36.912809 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.912776    2583 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:13:36.912860 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.912814    2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:13:36.912860 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.912832    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:13:36.913539 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.913528    2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:13:36.916972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.916960    2583 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:13:36.917098 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.917090    2583 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:13:36.917714 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.917706    2583 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:13:36.917752 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.917717    2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:13:36.917752 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.917728    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:13:36.917752 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.917736    2583 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:13:36.917752 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.917744    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:13:36.918055 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.918043    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:13:36.922383 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.922364    2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:13:36.927079 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.927067    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:13:36.927992 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.927979    2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:13:36.928045 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928010    2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:13:36.928173 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928164    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:13:36.928206 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928178    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:13:36.928206 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928184    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:13:36.928206 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928190    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:13:36.928206 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928195    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:13:36.928206 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928201    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:13:36.928700 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928690    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:13:36.928700 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928700    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:13:36.928762 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928707    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:13:36.928762 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928712    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:13:36.928762 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928720    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:13:36.928762 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.928729    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:13:36.929891 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.929882    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:13:36.929891 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.929891    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:13:36.933431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.933419    2583 server.go:1263] "Started kubelet"
Oct 29 22:13:36.933543 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.933517    2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:13:36.933917 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.933594    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:13:36.933917 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.933669    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:13:36.934148 ip-10-0-128-31 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:13:36.936803 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.936779    2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:13:36.938679 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.938661    2583 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:13:36.939535 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:36.939515    2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 29 22:13:36.940679 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.940666    2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:13:36.940739 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.940694    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:13:36.940800 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:36.940777    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Oct 29 22:13:36.941176 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.941158    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:13:36.941555 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.941532    2583 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:13:36.941555 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.941557    2583 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:13:36.941674 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.941628    2583 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:13:36.941674 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.941638    2583 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:13:36.941851 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.941832    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:13:36.941851 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.941850    2583 factory.go:55] Registering systemd factory
Oct 29 22:13:36.942019 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.941858    2583 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:13:36.942083 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.942064    2583 factory.go:153] Registering CRI-O factory
Oct 29 22:13:36.942083 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.942081    2583 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:13:36.942239 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.942099    2583 factory.go:103] Registering Raw factory
Oct 29 22:13:36.942239 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.942112    2583 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:13:36.943301 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.943281    2583 manager.go:319] Starting recovery of all containers
Oct 29 22:13:36.951959 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.951944    2583 manager.go:324] Recovery completed
Oct 29 22:13:36.958308 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.958292    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:36.959991 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.959970    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:36.960074 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.960007    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:36.960074 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.960021    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:36.960568 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.960551    2583 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:13:36.960568 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.960566    2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:13:36.960714 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.960574    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:36.960714 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.960580    2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:13:36.961642 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.961614    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:36.961642 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.961637    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:36.962761 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.961693    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:36.962847 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.962805    2583 policy_none.go:50] "Start"
Oct 29 22:13:36.962847 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.962818    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:13:36.962847 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.962828    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:13:36.964907 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.964880    2583 policy_none.go:44] "Start"
Oct 29 22:13:36.969227 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:36.966118    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18e31fc72cc105c4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-10-29 22:13:36.933389764 +0000 UTC m=+0.332821025,LastTimestamp:2026-10-29 22:13:36.933389764 +0000 UTC m=+0.332821025,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}"
Oct 29 22:13:36.981261 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.981236    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:13:36.982687 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:36.982668    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-31.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 29 22:13:36.982767 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:36.982611    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18e31fc72e56e856  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-31.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-10-29 22:13:36.959989846 +0000 UTC m=+0.359421108,LastTimestamp:2026-10-29 22:13:36.959989846 +0000 UTC m=+0.359421108,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}"
Oct 29 22:13:36.982848 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:36.982825    2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 29 22:13:36.982954 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:36.982936    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 29 22:13:36.993815 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:36.993794    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 29 22:13:37.000359 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.000339    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pf5l2"
Oct 29 22:13:37.005941 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.005907    2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:13:37.007154 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.007138    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pf5l2"
Oct 29 22:13:37.007812 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.007791    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 29 22:13:37.007952 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.007895    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-31.ec2.internal.18e31fc72e57453e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-31.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-31.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-31.ec2.internal,},FirstTimestamp:2026-10-29 22:13:36.96001363 +0000 UTC m=+0.359444893,LastTimestamp:2026-10-29 22:13:36.96001363 +0000 UTC m=+0.359444893,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-31.ec2.internal,}"
Oct 29 22:13:37.008954 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.008941    2583 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:13:37.009037 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.008966    2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:13:37.009037 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.008974    2583 server.go:87] "Starting device plugin registration server"
Oct 29 22:13:37.009213 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.009201    2583 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:13:37.009249 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.009214    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:13:37.009351 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.009334    2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:13:37.009438 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.009408    2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:13:37.009438 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.009415    2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:13:37.009838 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.009819    2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:13:37.009902 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.009867    2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-31.ec2.internal\" not found"
Oct 29 22:13:37.080545 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.080514    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:13:37.082511 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.081696    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:13:37.082511 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.081714    2583 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:13:37.082511 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.081732    2583 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:13:37.082511 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.081766    2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 29 22:13:37.098445 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.098430    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:37.110002 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.109983    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:37.110837 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.110822    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:37.110915 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.110849    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:37.110915 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.110861    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:37.110915 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.110882    2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.130642 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.130591    2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.130642 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.130611    2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-31.ec2.internal\": node \"ip-10-0-128-31.ec2.internal\" not found"
Oct 29 22:13:37.173629 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.173611    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Oct 29 22:13:37.182657 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.182638    2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"]
Oct 29 22:13:37.182704 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.182686    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:37.184000 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.183986    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:37.184059 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.184013    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:37.184059 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.184028    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:37.186239 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.186227    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:37.186417 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.186402    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.186458 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.186431    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:37.188382 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.188366    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:37.188382 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.188374    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:37.188508 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.188395    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:37.188508 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.188399    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:37.188508 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.188410    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:37.188606 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.188410    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:37.190663 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.190646    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.190761 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.190671    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:13:37.191508 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.191492    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:13:37.191590 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.191513    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:13:37.191590 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.191522    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:13:37.208799 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.208782    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-31.ec2.internal\" not found" node="ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.214222 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.214203    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-31.ec2.internal\" not found" node="ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.243264 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.243242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.243358 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.243271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.243358 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.243288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed322e80a89b2bbeafbf0f88fe9597d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"ed322e80a89b2bbeafbf0f88fe9597d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.274099 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.274081    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Oct 29 22:13:37.343971 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.343952    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.344057 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.344014    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.344057 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.344047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.344147 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.344062    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.344147 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.344091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed322e80a89b2bbeafbf0f88fe9597d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"ed322e80a89b2bbeafbf0f88fe9597d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.344147 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.344124    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed322e80a89b2bbeafbf0f88fe9597d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"ed322e80a89b2bbeafbf0f88fe9597d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.374802 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.374780    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Oct 29 22:13:37.475060 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.475012    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Oct 29 22:13:37.512205 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.512180    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.517669 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.517655    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.575523 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.575502    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Oct 29 22:13:37.675966 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.675938    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Oct 29 22:13:37.776503 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.776455    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Oct 29 22:13:37.825155 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.825137    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:37.877521 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:37.877492    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Oct 29 22:13:37.881656 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.881641    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 29 22:13:37.881765 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.881745    2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 22:13:37.881809 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.881790    2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 22:13:37.939049 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.938912    2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:37.940955 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.940935    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 29 22:13:37.941061 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.940941    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.962474 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.962458    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:13:37.964027 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.964004    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:13:37.989730 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.989708    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:13:37.989943 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:37.989924    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:13:38.011448 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.011413    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:08:37 +0000 UTC" deadline="2028-07-04 10:32:58.287346756 +0000 UTC"
Oct 29 22:13:38.011448 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.011450    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14724h19m20.275902676s"
Oct 29 22:13:38.053230 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.053185    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d7z9j"
Oct 29 22:13:38.061494 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.061475    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d7z9j"
Oct 29 22:13:38.078096 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:38.078069    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded322e80a89b2bbeafbf0f88fe9597d6.slice/crio-5ca54851dd320d460ba4b60f33b7050ac1630d97a10d399d90c6362a80b98572 WatchSource:0}: Error finding container 5ca54851dd320d460ba4b60f33b7050ac1630d97a10d399d90c6362a80b98572: Status 404 returned error can't find the container with id 5ca54851dd320d460ba4b60f33b7050ac1630d97a10d399d90c6362a80b98572
Oct 29 22:13:38.078324 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:38.078309    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb6269644393a710c6775a446a340ab.slice/crio-2491f2b3af47d9d9e0dd4179ca3d0ecffc898997a55367d2ff090a61eccfbbc7 WatchSource:0}: Error finding container 2491f2b3af47d9d9e0dd4179ca3d0ecffc898997a55367d2ff090a61eccfbbc7: Status 404 returned error can't find the container with id 2491f2b3af47d9d9e0dd4179ca3d0ecffc898997a55367d2ff090a61eccfbbc7
Oct 29 22:13:38.084164 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.084150    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:13:38.084825 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.084780    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" event={"ID":"ed322e80a89b2bbeafbf0f88fe9597d6","Type":"ContainerStarted","Data":"5ca54851dd320d460ba4b60f33b7050ac1630d97a10d399d90c6362a80b98572"}
Oct 29 22:13:38.086149 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.086130    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"1bb6269644393a710c6775a446a340ab","Type":"ContainerStarted","Data":"2491f2b3af47d9d9e0dd4179ca3d0ecffc898997a55367d2ff090a61eccfbbc7"}
Oct 29 22:13:38.505025 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.504949    2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:38.919569 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.919486    2583 apiserver.go:52] "Watching apiserver"
Oct 29 22:13:38.948115 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.948084    2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:13:38.948541 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.948508    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-sd9mh","openshift-network-operator/iptables-alerter-7lbzl","kube-system/konnectivity-agent-99lmd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5","openshift-cluster-node-tuning-operator/tuned-mmk2r","openshift-multus/network-metrics-daemon-l9rds","kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal","openshift-dns/node-resolver-m7gqm","openshift-image-registry/node-ca-x8qdl","openshift-ovn-kubernetes/ovnkube-node-774ht","kube-system/global-pull-secret-syncer-stxld","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal","openshift-multus/multus-75pjw","openshift-multus/multus-additional-cni-plugins-xqtzn"]
Oct 29 22:13:38.951332 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.951308    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:38.951467 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:38.951417    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:13:38.953754 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.953730    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:13:38.956121 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.956101    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:13:38.958996 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.958977    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:38.961772 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.961752    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:38.964005 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.963983    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:38.964119 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:38.964061    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:13:38.966744 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.966615    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:13:38.969098 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.969073    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:13:38.969197 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.969116    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:13:38.969197 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.969140    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:13:38.969270 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.969210    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f584j\""
Oct 29 22:13:38.969684 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.969664    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:13:38.970530 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.970499    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cdnlg\""
Oct 29 22:13:38.972713 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.972659    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:38.974505 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.974474    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:13:38.974505 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.974487    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:13:38.974707 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.974557    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8zjwp\""
Oct 29 22:13:38.974707 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.974612    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:13:38.974707 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.974621    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:13:38.974707 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.974640    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:13:38.975696 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.975358    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:38.975696 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:38.975424    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:13:38.976045 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.976025    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bjh7c\""
Oct 29 22:13:38.976140 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.976055    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:13:38.976235 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.976222    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:13:38.979009 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.978434    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-75pjw"
Oct 29 22:13:38.981590 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.981215    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:13:38.981590 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.981326    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:38.981590 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.981350    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kr68l\""
Oct 29 22:13:38.981590 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.981451    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:13:38.987257 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:38.987238    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:13:39.042505 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.042483    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:13:39.053574 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c8fff1-c08b-4437-944f-9ee33d186c42-agent-certs\") pod \"konnectivity-agent-99lmd\" (UID: \"b6c8fff1-c08b-4437-944f-9ee33d186c42\") " pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:13:39.053677 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c8fff1-c08b-4437-944f-9ee33d186c42-konnectivity-ca\") pod \"konnectivity-agent-99lmd\" (UID: \"b6c8fff1-c08b-4437-944f-9ee33d186c42\") " pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:13:39.053677 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:39.053677 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-kubelet\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.053833 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.053833 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-cni-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.053923 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-etc-kubernetes\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.053923 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.053923 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-etc-selinux\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.053923 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053906    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-slash\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.054081 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053925    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-host\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.054081 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-tuned\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.054081 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d56328b-9fa0-45cb-996f-65c1c1b8785c-dbus\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:39.054081 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.053986    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-system-cni-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.054081 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-cni-bin\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.054081 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-registration-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.054081 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054048    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-sys-fs\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.054081 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054062    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-node-log\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.054081 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysctl-conf\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054106    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mht74\" (UniqueName: \"kubernetes.io/projected/e21cb68b-ed42-4009-9bf6-9700a98ffd42-kube-api-access-mht74\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054170    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrmzq\" (UniqueName: \"kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq\") pod \"network-check-target-sd9mh\" (UID: \"5345c4bc-ec8a-454b-83bf-363e0d1432e6\") " pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054185    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-run\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054241    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjgg2\" (UniqueName: \"kubernetes.io/projected/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-kube-api-access-zjgg2\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-conf-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-cnibin\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-systemd\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-tmp\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054385    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhfw\" (UniqueName: \"kubernetes.io/projected/0605dc25-2004-42f2-a856-2f0be8e75ca2-kube-api-access-hkhfw\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-log-socket\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-cni-bin\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.054465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054470    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-var-lib-kubelet\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054493    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d56328b-9fa0-45cb-996f-65c1c1b8785c-kubelet-config\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qdlv\" (UniqueName: \"kubernetes.io/projected/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-kube-api-access-6qdlv\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054524    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93d782aa-ad96-4f81-a5a1-d356d022106b-host\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054539    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93d782aa-ad96-4f81-a5a1-d356d022106b-serviceca\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4c49406-6e72-4bdd-8375-20dffea48c35-hosts-file\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054576    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-modprobe-d\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rgp\" (UniqueName: \"kubernetes.io/projected/a4c49406-6e72-4bdd-8375-20dffea48c35-kube-api-access-68rgp\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-socket-dir-parent\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-k8s-cni-cncf-io\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7drt\" (UniqueName: \"kubernetes.io/projected/93d782aa-ad96-4f81-a5a1-d356d022106b-kube-api-access-d7drt\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-kubelet\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054713    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-var-lib-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-run-netns\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054748    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-run-ovn-kubernetes\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054763    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76cvd\" (UniqueName: \"kubernetes.io/projected/eb9f947e-2e34-4b91-9e2b-1edafb918fef-kube-api-access-76cvd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.054972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c49406-6e72-4bdd-8375-20dffea48c35-tmp-dir\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e0ef4de-c402-4c07-92f9-a7ec7d263113-cni-binary-copy\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-hostroot\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-multus-certs\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-systemd-units\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-cni-netd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054889    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovnkube-config\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054921    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-env-overrides\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovnkube-script-lib\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-sys\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054965    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-cnibin\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.054994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-systemd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-ovn\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055049    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovn-node-metrics-cert\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysconfig\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-cni-multus\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.055662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g29f\" (UniqueName: \"kubernetes.io/projected/8e0ef4de-c402-4c07-92f9-a7ec7d263113-kube-api-access-2g29f\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk9z5\" (UniqueName: \"kubernetes.io/projected/3611f6fc-6112-4b2e-8e22-75672c63c673-kube-api-access-qk9z5\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-socket-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysctl-d\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-lib-modules\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-system-cni-dir\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-cni-binary-copy\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055289    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e21cb68b-ed42-4009-9bf6-9700a98ffd42-iptables-alerter-script\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e21cb68b-ed42-4009-9bf6-9700a98ffd42-host-slash\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055328    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-device-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055356    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-etc-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-kubernetes\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-netns\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.056431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.055711    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-daemon-config\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.065203 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.065146    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:08:39 +0000 UTC" deadline="2028-05-30 04:58:10.421269617 +0000 UTC"
Oct 29 22:13:39.065203 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.065198    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13878h44m31.356074771s"
Oct 29 22:13:39.077914 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.077894    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fz5pc\""
Oct 29 22:13:39.078015 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.077970    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p5dkc\""
Oct 29 22:13:39.078183 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.078161    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:13:39.078295 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.078197    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:13:39.078295 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.078249    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:13:39.078295 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.078287    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:13:39.078478 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.078339    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:13:39.078632 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.078616    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:13:39.078703 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.078691    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:13:39.078763 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.078729    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:13:39.078825 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.078809    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:13:39.078903 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.078889    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:13:39.079010 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.078989    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bwm9d\""
Oct 29 22:13:39.079117 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.079095    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:13:39.079186 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.079173    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:13:39.079289 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.079274    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:13:39.079552 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.079527    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:13:39.079632 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.079575    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mwjl8\""
Oct 29 22:13:39.156544 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmzq\" (UniqueName: \"kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq\") pod \"network-check-target-sd9mh\" (UID: \"5345c4bc-ec8a-454b-83bf-363e0d1432e6\") " pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:39.156544 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-run\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.156771 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgg2\" (UniqueName: \"kubernetes.io/projected/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-kube-api-access-zjgg2\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.156771 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-conf-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.156771 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-cnibin\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.156771 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.156771 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156678    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-systemd\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.156771 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-tmp\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.156771 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhfw\" (UniqueName: \"kubernetes.io/projected/0605dc25-2004-42f2-a856-2f0be8e75ca2-kube-api-access-hkhfw\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.156771 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-log-socket\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-cni-bin\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-var-lib-kubelet\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d56328b-9fa0-45cb-996f-65c1c1b8785c-kubelet-config\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156828    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-run\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qdlv\" (UniqueName: \"kubernetes.io/projected/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-kube-api-access-6qdlv\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156870    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93d782aa-ad96-4f81-a5a1-d356d022106b-host\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156900    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93d782aa-ad96-4f81-a5a1-d356d022106b-serviceca\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156911    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-conf-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156922    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-cnibin\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156967    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-systemd\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156975    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4c49406-6e72-4bdd-8375-20dffea48c35-hosts-file\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.156926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4c49406-6e72-4bdd-8375-20dffea48c35-hosts-file\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:13:39.157016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-modprobe-d\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rgp\" (UniqueName: \"kubernetes.io/projected/a4c49406-6e72-4bdd-8375-20dffea48c35-kube-api-access-68rgp\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-socket-dir-parent\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-k8s-cni-cncf-io\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157095    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7drt\" (UniqueName: \"kubernetes.io/projected/93d782aa-ad96-4f81-a5a1-d356d022106b-kube-api-access-d7drt\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157112    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-kubelet\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157114    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-log-socket\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-var-lib-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-run-netns\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157164    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-cni-bin\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157212    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-socket-dir-parent\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157224    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-var-lib-kubelet\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157237    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-run-ovn-kubernetes\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157266    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-k8s-cni-cncf-io\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.157405 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157275    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d56328b-9fa0-45cb-996f-65c1c1b8785c-kubelet-config\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-run-ovn-kubernetes\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cvd\" (UniqueName: \"kubernetes.io/projected/eb9f947e-2e34-4b91-9e2b-1edafb918fef-kube-api-access-76cvd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c49406-6e72-4bdd-8375-20dffea48c35-tmp-dir\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e0ef4de-c402-4c07-92f9-a7ec7d263113-cni-binary-copy\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157525    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-kubelet\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-hostroot\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-multus-certs\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-systemd-units\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157583    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-var-lib-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-cni-netd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157631    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-run-netns\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovnkube-config\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-env-overrides\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157683    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-modprobe-d\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovnkube-script-lib\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-sys\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.158020 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157741    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-cnibin\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-systemd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-ovn\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovn-node-metrics-cert\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysconfig\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-cni-multus\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157864    2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g29f\" (UniqueName: \"kubernetes.io/projected/8e0ef4de-c402-4c07-92f9-a7ec7d263113-kube-api-access-2g29f\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk9z5\" (UniqueName: \"kubernetes.io/projected/3611f6fc-6112-4b2e-8e22-75672c63c673-kube-api-access-qk9z5\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-socket-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysctl-d\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-lib-modules\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158020    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-system-cni-dir\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-cni-binary-copy\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e21cb68b-ed42-4009-9bf6-9700a98ffd42-iptables-alerter-script\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e21cb68b-ed42-4009-9bf6-9700a98ffd42-host-slash\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:13:39.158688 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-device-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158143    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-systemd-units\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-etc-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-kubernetes\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158196    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-netns\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158257    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-daemon-config\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c8fff1-c08b-4437-944f-9ee33d186c42-agent-certs\") pod \"konnectivity-agent-99lmd\" (UID: \"b6c8fff1-c08b-4437-944f-9ee33d186c42\") " pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158279    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-socket-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c8fff1-c08b-4437-944f-9ee33d186c42-konnectivity-ca\") pod \"konnectivity-agent-99lmd\" (UID: \"b6c8fff1-c08b-4437-944f-9ee33d186c42\") " pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158304    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c49406-6e72-4bdd-8375-20dffea48c35-tmp-dir\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158313    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158352    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-kubelet\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.158387    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158432    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysctl-d\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.159362 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-cni-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.158489    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs podName:a97d96fb-da2b-4064-b6bb-ac7f3cf2270b nodeName:}" failed. No retries permitted until 2026-10-29 22:13:39.65844985 +0000 UTC m=+3.057881123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs") pod "network-metrics-daemon-l9rds" (UID: "a97d96fb-da2b-4064-b6bb-ac7f3cf2270b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158487    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-system-cni-dir\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158272    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93d782aa-ad96-4f81-a5a1-d356d022106b-host\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158487    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-etc-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158048    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93d782aa-ad96-4f81-a5a1-d356d022106b-serviceca\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158522    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-cni-netd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158565    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-cni-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158612    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-device-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158614    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-lib-modules\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158888    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovnkube-config\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158952    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-env-overrides\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158997    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158080    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-multus-certs\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159208    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c8fff1-c08b-4437-944f-9ee33d186c42-konnectivity-ca\") pod \"konnectivity-agent-99lmd\" (UID: \"b6c8fff1-c08b-4437-944f-9ee33d186c42\") " pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159318    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-kubernetes\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159459    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e0ef4de-c402-4c07-92f9-a7ec7d263113-cni-binary-copy\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.159940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159514    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-hostroot\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.158305    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e21cb68b-ed42-4009-9bf6-9700a98ffd42-host-slash\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159566    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysconfig\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159596    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e21cb68b-ed42-4009-9bf6-9700a98ffd42-iptables-alerter-script\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159662    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-daemon-config\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159658    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-sys\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.159690    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159718    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-cnibin\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159735    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-netns\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.159743    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret podName:7d56328b-9fa0-45cb-996f-65c1c1b8785c nodeName:}" failed. No retries permitted until 2026-10-29 22:13:39.659726822 +0000 UTC m=+3.059158090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret") pod "global-pull-secret-syncer-stxld" (UID: "7d56328b-9fa0-45cb-996f-65c1c1b8785c") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159770    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-systemd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-etc-kubernetes\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159804    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-cni-binary-copy\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159817    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-ovn\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-etc-selinux\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159870    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-etc-kubernetes\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.160542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-slash\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-host\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159919    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-slash\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-tuned\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159967    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d56328b-9fa0-45cb-996f-65c1c1b8785c-dbus\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.159990    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-cni-multus\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-system-cni-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-cni-bin\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160045    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-etc-selinux\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-registration-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160086    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-cni-bin\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-sys-fs\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-node-log\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160156    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysctl-conf\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160222    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mht74\" (UniqueName: \"kubernetes.io/projected/e21cb68b-ed42-4009-9bf6-9700a98ffd42-kube-api-access-mht74\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:13:39.161234 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160435    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-system-cni-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.162144 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160416    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d56328b-9fa0-45cb-996f-65c1c1b8785c-dbus\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:39.162144 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.157876    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.162144 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160538    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-sys-fs\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.162144 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160532    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-host\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.162144 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160597    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-node-log\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.162144 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160614    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-registration-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.162144 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160599    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-kubelet\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.162144 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160662    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.162144 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160747    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysctl-conf\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.162144 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.160774    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.162144 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.161305    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovnkube-script-lib\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.162684 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.162558    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-tuned\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.162684 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.162628    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c8fff1-c08b-4437-944f-9ee33d186c42-agent-certs\") pod \"konnectivity-agent-99lmd\" (UID: \"b6c8fff1-c08b-4437-944f-9ee33d186c42\") " pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:13:39.163721 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.163695    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovn-node-metrics-cert\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.163845 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.163821    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-tmp\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.178642 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.178578    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhfw\" (UniqueName: \"kubernetes.io/projected/0605dc25-2004-42f2-a856-2f0be8e75ca2-kube-api-access-hkhfw\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.179261 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.179232    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:39.179261 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.179259    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:39.179447 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.179273    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rrmzq for pod openshift-network-diagnostics/network-check-target-sd9mh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:39.179447 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.179363    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq podName:5345c4bc-ec8a-454b-83bf-363e0d1432e6 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:39.679333214 +0000 UTC m=+3.078764480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rrmzq" (UniqueName: "kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq") pod "network-check-target-sd9mh" (UID: "5345c4bc-ec8a-454b-83bf-363e0d1432e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:39.181734 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.181708    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7drt\" (UniqueName: \"kubernetes.io/projected/93d782aa-ad96-4f81-a5a1-d356d022106b-kube-api-access-d7drt\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:13:39.182280 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.182185    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjgg2\" (UniqueName: \"kubernetes.io/projected/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-kube-api-access-zjgg2\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.184291 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.184135    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cvd\" (UniqueName: \"kubernetes.io/projected/eb9f947e-2e34-4b91-9e2b-1edafb918fef-kube-api-access-76cvd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.185537 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.185505    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mht74\" (UniqueName: \"kubernetes.io/projected/e21cb68b-ed42-4009-9bf6-9700a98ffd42-kube-api-access-mht74\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:13:39.185628 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.185572    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk9z5\" (UniqueName: \"kubernetes.io/projected/3611f6fc-6112-4b2e-8e22-75672c63c673-kube-api-access-qk9z5\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.185692 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.185621    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qdlv\" (UniqueName: \"kubernetes.io/projected/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-kube-api-access-6qdlv\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:39.187590 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.186509    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rgp\" (UniqueName: \"kubernetes.io/projected/a4c49406-6e72-4bdd-8375-20dffea48c35-kube-api-access-68rgp\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:13:39.189434 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.189368    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g29f\" (UniqueName: \"kubernetes.io/projected/8e0ef4de-c402-4c07-92f9-a7ec7d263113-kube-api-access-2g29f\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.269123 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.269091    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:13:39.279554 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.279521    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:13:39.282058 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:39.282018    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21cb68b_ed42_4009_9bf6_9700a98ffd42.slice/crio-e2c7eecb67826582f9af117bab16905f4d8e24ea31d610a970e65b890b2f90b7 WatchSource:0}: Error finding container e2c7eecb67826582f9af117bab16905f4d8e24ea31d610a970e65b890b2f90b7: Status 404 returned error can't find the container with id e2c7eecb67826582f9af117bab16905f4d8e24ea31d610a970e65b890b2f90b7
Oct 29 22:13:39.291314 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:39.291289    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c8fff1_c08b_4437_944f_9ee33d186c42.slice/crio-7340e678fa66b915b570522eda264d7ee075877a3034baafcbf53d9c53a82a81 WatchSource:0}: Error finding container 7340e678fa66b915b570522eda264d7ee075877a3034baafcbf53d9c53a82a81: Status 404 returned error can't find the container with id 7340e678fa66b915b570522eda264d7ee075877a3034baafcbf53d9c53a82a81
Oct 29 22:13:39.293961 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.293941    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:13:39.295301 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.295249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:13:39.302881 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.302855    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:13:39.308410 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:39.308364    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c23cb73_3b67_46d7_94ab_ad2c0d9473de.slice/crio-293285400af1fb68dff3126aba450395f7000efc20254d0c5e291898485460ca WatchSource:0}: Error finding container 293285400af1fb68dff3126aba450395f7000efc20254d0c5e291898485460ca: Status 404 returned error can't find the container with id 293285400af1fb68dff3126aba450395f7000efc20254d0c5e291898485460ca
Oct 29 22:13:39.309603 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:39.309578    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0605dc25_2004_42f2_a856_2f0be8e75ca2.slice/crio-e228425838c767c3d51e8a7849ae3082ef6bc00e11ba60b2058d289ba906a452 WatchSource:0}: Error finding container e228425838c767c3d51e8a7849ae3082ef6bc00e11ba60b2058d289ba906a452: Status 404 returned error can't find the container with id e228425838c767c3d51e8a7849ae3082ef6bc00e11ba60b2058d289ba906a452
Oct 29 22:13:39.310428 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.310402    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:13:39.318917 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:39.318897    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c49406_6e72_4bdd_8375_20dffea48c35.slice/crio-3d13fbd069491dc9e4bf3dc0076510878209847c2817f6c67275947f9e146fe3 WatchSource:0}: Error finding container 3d13fbd069491dc9e4bf3dc0076510878209847c2817f6c67275947f9e146fe3: Status 404 returned error can't find the container with id 3d13fbd069491dc9e4bf3dc0076510878209847c2817f6c67275947f9e146fe3
Oct 29 22:13:39.321883 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:39.321859    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d782aa_ad96_4f81_a5a1_d356d022106b.slice/crio-3a92538a2ac695ed07c930ed2920e9c8efde421eb38e43c4f231f4a2dd42963d WatchSource:0}: Error finding container 3a92538a2ac695ed07c930ed2920e9c8efde421eb38e43c4f231f4a2dd42963d: Status 404 returned error can't find the container with id 3a92538a2ac695ed07c930ed2920e9c8efde421eb38e43c4f231f4a2dd42963d
Oct 29 22:13:39.331519 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.331481    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-75pjw"
Oct 29 22:13:39.334603 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.334580    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:39.342018 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.341995    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:39.342552 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:39.342510    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0ef4de_c402_4c07_92f9_a7ec7d263113.slice/crio-7bdf80363cc8ebc6e87f026eb86af0a1a09217f9a7e1bd59ec015198b9f5b1fd WatchSource:0}: Error finding container 7bdf80363cc8ebc6e87f026eb86af0a1a09217f9a7e1bd59ec015198b9f5b1fd: Status 404 returned error can't find the container with id 7bdf80363cc8ebc6e87f026eb86af0a1a09217f9a7e1bd59ec015198b9f5b1fd
Oct 29 22:13:39.351954 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:39.351934    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9f947e_2e34_4b91_9e2b_1edafb918fef.slice/crio-38c03c40eb74d96c3e261aa2044718a84fc65b51bcf5b776c0419a5faead95ec WatchSource:0}: Error finding container 38c03c40eb74d96c3e261aa2044718a84fc65b51bcf5b776c0419a5faead95ec: Status 404 returned error can't find the container with id 38c03c40eb74d96c3e261aa2044718a84fc65b51bcf5b776c0419a5faead95ec
Oct 29 22:13:39.352662 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.352635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:13:39.363097 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:13:39.363075    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3611f6fc_6112_4b2e_8e22_75672c63c673.slice/crio-b807268e76cb9ef3e16b3b00e571b3334acfba840b9008af81982896d2ccabbe WatchSource:0}: Error finding container b807268e76cb9ef3e16b3b00e571b3334acfba840b9008af81982896d2ccabbe: Status 404 returned error can't find the container with id b807268e76cb9ef3e16b3b00e571b3334acfba840b9008af81982896d2ccabbe
Oct 29 22:13:39.664741 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.664713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:39.664832 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.664758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:39.664894 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.664879    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:39.664956 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.664879    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:39.664956 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.664939    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs podName:a97d96fb-da2b-4064-b6bb-ac7f3cf2270b nodeName:}" failed. No retries permitted until 2026-10-29 22:13:40.664926162 +0000 UTC m=+4.064357413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs") pod "network-metrics-daemon-l9rds" (UID: "a97d96fb-da2b-4064-b6bb-ac7f3cf2270b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:39.665054 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.665013    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret podName:7d56328b-9fa0-45cb-996f-65c1c1b8785c nodeName:}" failed. No retries permitted until 2026-10-29 22:13:40.664994406 +0000 UTC m=+4.064425661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret") pod "global-pull-secret-syncer-stxld" (UID: "7d56328b-9fa0-45cb-996f-65c1c1b8785c") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:39.765603 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:39.765576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmzq\" (UniqueName: \"kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq\") pod \"network-check-target-sd9mh\" (UID: \"5345c4bc-ec8a-454b-83bf-363e0d1432e6\") " pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:39.765775 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.765758    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:39.765848 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.765785    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:39.765848 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.765798    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rrmzq for pod openshift-network-diagnostics/network-check-target-sd9mh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:39.765948 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:39.765851    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq podName:5345c4bc-ec8a-454b-83bf-363e0d1432e6 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:40.765832906 +0000 UTC m=+4.165264171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrmzq" (UniqueName: "kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq") pod "network-check-target-sd9mh" (UID: "5345c4bc-ec8a-454b-83bf-363e0d1432e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:40.065657 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.065538    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:08:39 +0000 UTC" deadline="2028-07-19 20:00:22.708805102 +0000 UTC"
Oct 29 22:13:40.065657 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.065606    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15093h46m42.643204099s"
Oct 29 22:13:40.083074 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.082548    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:40.083074 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:40.082685    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:13:40.108501 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.108390    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-774ht" event={"ID":"eb9f947e-2e34-4b91-9e2b-1edafb918fef","Type":"ContainerStarted","Data":"38c03c40eb74d96c3e261aa2044718a84fc65b51bcf5b776c0419a5faead95ec"}
Oct 29 22:13:40.115257 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.115183    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-75pjw" event={"ID":"8e0ef4de-c402-4c07-92f9-a7ec7d263113","Type":"ContainerStarted","Data":"7bdf80363cc8ebc6e87f026eb86af0a1a09217f9a7e1bd59ec015198b9f5b1fd"}
Oct 29 22:13:40.142596 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.142567    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x8qdl" event={"ID":"93d782aa-ad96-4f81-a5a1-d356d022106b","Type":"ContainerStarted","Data":"3a92538a2ac695ed07c930ed2920e9c8efde421eb38e43c4f231f4a2dd42963d"}
Oct 29 22:13:40.154286 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.154258    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mmk2r" event={"ID":"5c23cb73-3b67-46d7-94ab-ad2c0d9473de","Type":"ContainerStarted","Data":"293285400af1fb68dff3126aba450395f7000efc20254d0c5e291898485460ca"}
Oct 29 22:13:40.160026 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.160006    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:13:40.188419 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.165677    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7lbzl" event={"ID":"e21cb68b-ed42-4009-9bf6-9700a98ffd42","Type":"ContainerStarted","Data":"e2c7eecb67826582f9af117bab16905f4d8e24ea31d610a970e65b890b2f90b7"}
Oct 29 22:13:40.191162 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.190028    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" event={"ID":"ed322e80a89b2bbeafbf0f88fe9597d6","Type":"ContainerStarted","Data":"e9f2c0851e694203096b6731d4cacaa871aaa693db1fe52f6ffcdbd335d2e49e"}
Oct 29 22:13:40.195052 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.193960    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xqtzn" event={"ID":"3611f6fc-6112-4b2e-8e22-75672c63c673","Type":"ContainerStarted","Data":"b807268e76cb9ef3e16b3b00e571b3334acfba840b9008af81982896d2ccabbe"}
Oct 29 22:13:40.196582 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.196534    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m7gqm" event={"ID":"a4c49406-6e72-4bdd-8375-20dffea48c35","Type":"ContainerStarted","Data":"3d13fbd069491dc9e4bf3dc0076510878209847c2817f6c67275947f9e146fe3"}
Oct 29 22:13:40.203190 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.203074    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5" event={"ID":"0605dc25-2004-42f2-a856-2f0be8e75ca2","Type":"ContainerStarted","Data":"e228425838c767c3d51e8a7849ae3082ef6bc00e11ba60b2058d289ba906a452"}
Oct 29 22:13:40.220310 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.220282    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-99lmd" event={"ID":"b6c8fff1-c08b-4437-944f-9ee33d186c42","Type":"ContainerStarted","Data":"7340e678fa66b915b570522eda264d7ee075877a3034baafcbf53d9c53a82a81"}
Oct 29 22:13:40.224049 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.224000    2583 generic.go:356] "Generic (PLEG): container finished" podID="1bb6269644393a710c6775a446a340ab" containerID="1acf379624438ddd0be82d11391dfc18fd2291e74ca790f56bcc8c92bc7d3247" exitCode=0
Oct 29 22:13:40.224049 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.224034    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"1bb6269644393a710c6775a446a340ab","Type":"ContainerDied","Data":"1acf379624438ddd0be82d11391dfc18fd2291e74ca790f56bcc8c92bc7d3247"}
Oct 29 22:13:40.271977 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.271905    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" podStartSLOduration=2.271888567 podStartE2EDuration="2.271888567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:40.231281645 +0000 UTC m=+3.630712903" watchObservedRunningTime="2026-10-29 22:13:40.271888567 +0000 UTC m=+3.671319844"
Oct 29 22:13:40.680608 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.679484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:40.680608 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.679783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:40.680608 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:40.679964    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:40.680608 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:40.680031    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs podName:a97d96fb-da2b-4064-b6bb-ac7f3cf2270b nodeName:}" failed. No retries permitted until 2026-10-29 22:13:42.680009051 +0000 UTC m=+6.079440324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs") pod "network-metrics-daemon-l9rds" (UID: "a97d96fb-da2b-4064-b6bb-ac7f3cf2270b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:40.680608 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:40.680466    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:40.680608 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:40.680523    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret podName:7d56328b-9fa0-45cb-996f-65c1c1b8785c nodeName:}" failed. No retries permitted until 2026-10-29 22:13:42.680506496 +0000 UTC m=+6.079937760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret") pod "global-pull-secret-syncer-stxld" (UID: "7d56328b-9fa0-45cb-996f-65c1c1b8785c") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:40.781979 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:40.781242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmzq\" (UniqueName: \"kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq\") pod \"network-check-target-sd9mh\" (UID: \"5345c4bc-ec8a-454b-83bf-363e0d1432e6\") " pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:40.781979 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:40.781455    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:40.781979 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:40.781481    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:40.781979 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:40.781495    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rrmzq for pod openshift-network-diagnostics/network-check-target-sd9mh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:40.781979 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:40.781558    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq podName:5345c4bc-ec8a-454b-83bf-363e0d1432e6 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:42.781538755 +0000 UTC m=+6.180970028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrmzq" (UniqueName: "kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq") pod "network-check-target-sd9mh" (UID: "5345c4bc-ec8a-454b-83bf-363e0d1432e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:41.085952 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:41.083237    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:41.085952 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:41.083369    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:13:41.085952 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:41.083676    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:41.085952 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:41.083752    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:13:41.244198 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:41.244159    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"1bb6269644393a710c6775a446a340ab","Type":"ContainerStarted","Data":"e04bfb33f6fe4adb6a7bcf38a1199ade15c8953b31cc293179bb69970bdd4b1a"}
Oct 29 22:13:42.083629 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:42.082625    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:42.083629 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:42.082745    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:13:42.258569 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:42.257725    2583 generic.go:356] "Generic (PLEG): container finished" podID="3611f6fc-6112-4b2e-8e22-75672c63c673" containerID="45bd423c6d556e36d2f1ec58c952847192bd659bd06d51fadccdd3e23e4417a3" exitCode=0
Oct 29 22:13:42.258569 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:42.258460    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xqtzn" event={"ID":"3611f6fc-6112-4b2e-8e22-75672c63c673","Type":"ContainerDied","Data":"45bd423c6d556e36d2f1ec58c952847192bd659bd06d51fadccdd3e23e4417a3"}
Oct 29 22:13:42.335576 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:42.334939    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" podStartSLOduration=4.334920246 podStartE2EDuration="4.334920246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:41.288915782 +0000 UTC m=+4.688347056" watchObservedRunningTime="2026-10-29 22:13:42.334920246 +0000 UTC m=+5.734351520"
Oct 29 22:13:42.694621 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:42.694494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:42.694621 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:42.694549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:42.694849 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:42.694667    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:42.694849 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:42.694735    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs podName:a97d96fb-da2b-4064-b6bb-ac7f3cf2270b nodeName:}" failed. No retries permitted until 2026-10-29 22:13:46.694714802 +0000 UTC m=+10.094146078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs") pod "network-metrics-daemon-l9rds" (UID: "a97d96fb-da2b-4064-b6bb-ac7f3cf2270b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:42.694849 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:42.694667    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:42.694849 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:42.694799    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret podName:7d56328b-9fa0-45cb-996f-65c1c1b8785c nodeName:}" failed. No retries permitted until 2026-10-29 22:13:46.694784567 +0000 UTC m=+10.094215832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret") pod "global-pull-secret-syncer-stxld" (UID: "7d56328b-9fa0-45cb-996f-65c1c1b8785c") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:42.795028 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:42.794984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmzq\" (UniqueName: \"kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq\") pod \"network-check-target-sd9mh\" (UID: \"5345c4bc-ec8a-454b-83bf-363e0d1432e6\") " pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:42.795210 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:42.795161    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:42.795210 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:42.795186    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:42.795210 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:42.795199    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rrmzq for pod openshift-network-diagnostics/network-check-target-sd9mh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:42.795440 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:42.795256    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq podName:5345c4bc-ec8a-454b-83bf-363e0d1432e6 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:46.795237941 +0000 UTC m=+10.194669205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrmzq" (UniqueName: "kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq") pod "network-check-target-sd9mh" (UID: "5345c4bc-ec8a-454b-83bf-363e0d1432e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:43.084762 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:43.084079    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:43.084762 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:43.084225    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:13:43.084762 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:43.084482    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:43.084762 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:43.084588    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:13:44.082604 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:44.082523    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:44.083020 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:44.082660    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:13:44.272389 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:44.272331    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-99lmd" event={"ID":"b6c8fff1-c08b-4437-944f-9ee33d186c42","Type":"ContainerStarted","Data":"009d10a7cfe47b55cd4a988fe69605f4b6ec2810decae3dc766b578c3db814cd"}
Oct 29 22:13:45.085468 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:45.085393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:45.085912 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:45.085560    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:13:45.086001 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:45.085979    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:45.086113 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:45.086091    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:13:45.278572 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:45.278500    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x8qdl" event={"ID":"93d782aa-ad96-4f81-a5a1-d356d022106b","Type":"ContainerStarted","Data":"2e24f787fe7ccda01f15336915281e7c467533aa4d8a26971f916cae4aedde99"}
Oct 29 22:13:45.322432 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:45.322363    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x8qdl" podStartSLOduration=2.313601609 podStartE2EDuration="7.322342173s" totalImagesPullingTime="5.008740564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:45.321772638 +0000 UTC m=+8.721203910" watchObservedRunningTime="2026-10-29 22:13:45.322342173 +0000 UTC m=+8.721773446"
Oct 29 22:13:45.323098 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:45.323060    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-99lmd" podStartSLOduration=2.908629041 podStartE2EDuration="7.323045815s" totalImagesPullingTime="4.414416774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:44.313515542 +0000 UTC m=+7.712946815" watchObservedRunningTime="2026-10-29 22:13:45.323045815 +0000 UTC m=+8.722477089"
Oct 29 22:13:46.082595 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:46.082557    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:46.082808 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:46.082698    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:13:46.284093 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:46.282646    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5" event={"ID":"0605dc25-2004-42f2-a856-2f0be8e75ca2","Type":"ContainerStarted","Data":"9733abcc6c76fb15f4ee8662a77c6804dd69a8548712999006c25c59229d7b2c"}
Oct 29 22:13:46.725907 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:46.725050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:46.725907 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:46.725109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:46.725907 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:46.725289    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:46.725907 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:46.725357    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs podName:a97d96fb-da2b-4064-b6bb-ac7f3cf2270b nodeName:}" failed. No retries permitted until 2026-10-29 22:13:54.725340318 +0000 UTC m=+18.124771592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs") pod "network-metrics-daemon-l9rds" (UID: "a97d96fb-da2b-4064-b6bb-ac7f3cf2270b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:46.725907 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:46.725801    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:46.725907 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:46.725869    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret podName:7d56328b-9fa0-45cb-996f-65c1c1b8785c nodeName:}" failed. No retries permitted until 2026-10-29 22:13:54.725852748 +0000 UTC m=+18.125284021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret") pod "global-pull-secret-syncer-stxld" (UID: "7d56328b-9fa0-45cb-996f-65c1c1b8785c") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:46.826612 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:46.825919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmzq\" (UniqueName: \"kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq\") pod \"network-check-target-sd9mh\" (UID: \"5345c4bc-ec8a-454b-83bf-363e0d1432e6\") " pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:46.826612 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:46.826131    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:46.826612 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:46.826151    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:46.826612 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:46.826164    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rrmzq for pod openshift-network-diagnostics/network-check-target-sd9mh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:46.826612 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:46.826222    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq podName:5345c4bc-ec8a-454b-83bf-363e0d1432e6 nodeName:}" failed. No retries permitted until 2026-10-29 22:13:54.826205045 +0000 UTC m=+18.225636310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrmzq" (UniqueName: "kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq") pod "network-check-target-sd9mh" (UID: "5345c4bc-ec8a-454b-83bf-363e0d1432e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:47.084449 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:47.084024    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:47.084449 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:47.084143    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:13:47.084449 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:47.084230    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:47.084449 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:47.084324    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:13:47.097947 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:47.097910    2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:13:47.290055 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:47.289918    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5" event={"ID":"0605dc25-2004-42f2-a856-2f0be8e75ca2","Type":"ContainerStarted","Data":"5f2460aaacd6a28ca7e0e7629d07f9f2f89a2758fc53409ee85c3a95f3a9669b"}
Oct 29 22:13:48.017901 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.017784    2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:13:47.097929931Z","UUID":"fa23f60c-2bb1-48c4-8d73-4b99a9a26732","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:13:48.019912 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.019543    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:13:48.019912 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.019573    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:13:48.082298 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.082262    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:48.082458 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:48.082412    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:13:48.085675 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.085643    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:13:48.086155 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.086133    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:13:48.294972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.294627    2583 generic.go:356] "Generic (PLEG): container finished" podID="3611f6fc-6112-4b2e-8e22-75672c63c673" containerID="c0059063d39f28606cf74561d0a69cc3d85ee5f009de816c01945234020cf868" exitCode=0
Oct 29 22:13:48.294972 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.294695    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xqtzn" event={"ID":"3611f6fc-6112-4b2e-8e22-75672c63c673","Type":"ContainerDied","Data":"c0059063d39f28606cf74561d0a69cc3d85ee5f009de816c01945234020cf868"}
Oct 29 22:13:48.297430 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.297313    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m7gqm" event={"ID":"a4c49406-6e72-4bdd-8375-20dffea48c35","Type":"ContainerStarted","Data":"29b5d40daa79d12567293d8a9fdab9f2700a83571ecb89a04b943033e896fc91"}
Oct 29 22:13:48.300838 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.300814    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5" event={"ID":"0605dc25-2004-42f2-a856-2f0be8e75ca2","Type":"ContainerStarted","Data":"b7a8daa73d200c5e77cd38b481bd68417f16c308bd9dd0e7c0554693d40f7152"}
Oct 29 22:13:48.303385 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.303354    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-75pjw" event={"ID":"8e0ef4de-c402-4c07-92f9-a7ec7d263113","Type":"ContainerStarted","Data":"2034756a7bcb841d1d5d1ce09d42848cd798bb9cf08e1aec0e75844d904ca7e3"}
Oct 29 22:13:48.303483 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.303405    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:13:48.304090 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.304071    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:13:48.386856 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.386810    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-75pjw" podStartSLOduration=2.134835354 podStartE2EDuration="10.386796419s" totalImagesPullingTime="8.251961065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:48.385042538 +0000 UTC m=+11.784473810" watchObservedRunningTime="2026-10-29 22:13:48.386796419 +0000 UTC m=+11.786227691"
Oct 29 22:13:48.464952 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.463992    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-m7gqm" podStartSLOduration=2.073999795 podStartE2EDuration="10.463973237s" totalImagesPullingTime="8.389973442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:48.463953638 +0000 UTC m=+11.863384907" watchObservedRunningTime="2026-10-29 22:13:48.463973237 +0000 UTC m=+11.863404509"
Oct 29 22:13:48.505670 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:48.505615    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5" podStartSLOduration=2.004627147 podStartE2EDuration="10.505602072s" totalImagesPullingTime="8.500974925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:48.504749022 +0000 UTC m=+11.904180325" watchObservedRunningTime="2026-10-29 22:13:48.505602072 +0000 UTC m=+11.905033347"
Oct 29 22:13:49.083326 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:49.082702    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:49.083326 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:49.082848    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:13:49.083326 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:49.083216    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:49.083326 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:49.083292    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:13:49.307575 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:49.307494    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7lbzl" event={"ID":"e21cb68b-ed42-4009-9bf6-9700a98ffd42","Type":"ContainerStarted","Data":"7875e5a8a91290067e1523369e46100661bc59888fd36b6377fa8913b20d3199"}
Oct 29 22:13:49.310546 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:49.310513    2583 generic.go:356] "Generic (PLEG): container finished" podID="3611f6fc-6112-4b2e-8e22-75672c63c673" containerID="96492b43253ec6d0147a13224a15ad4f71e8aed1940ff973e2199a7843615ce6" exitCode=0
Oct 29 22:13:49.310652 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:49.310616    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xqtzn" event={"ID":"3611f6fc-6112-4b2e-8e22-75672c63c673","Type":"ContainerDied","Data":"96492b43253ec6d0147a13224a15ad4f71e8aed1940ff973e2199a7843615ce6"}
Oct 29 22:13:49.387282 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:49.387229    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7lbzl" podStartSLOduration=3.189091342 podStartE2EDuration="11.387215356s" totalImagesPullingTime="8.198124014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:49.344910676 +0000 UTC m=+12.744341949" watchObservedRunningTime="2026-10-29 22:13:49.387215356 +0000 UTC m=+12.786646629"
Oct 29 22:13:50.082098 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:50.082072    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:50.082226 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:50.082194    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:13:50.314819 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:50.314788    2583 generic.go:356] "Generic (PLEG): container finished" podID="3611f6fc-6112-4b2e-8e22-75672c63c673" containerID="8a1ded9624a6c0d1e06e1c199299cdf59b65b0496034ee229c9698868b8f165a" exitCode=0
Oct 29 22:13:50.315698 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:50.314882    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xqtzn" event={"ID":"3611f6fc-6112-4b2e-8e22-75672c63c673","Type":"ContainerDied","Data":"8a1ded9624a6c0d1e06e1c199299cdf59b65b0496034ee229c9698868b8f165a"}
Oct 29 22:13:50.316645 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:50.316616    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mmk2r" event={"ID":"5c23cb73-3b67-46d7-94ab-ad2c0d9473de","Type":"ContainerStarted","Data":"d047833342fab11b4f3ea6ec4e1a8f26e44bee1f2717619705eeecf5d21c46f6"}
Oct 29 22:13:50.426038 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:50.425934    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mmk2r" podStartSLOduration=2.02116796 podStartE2EDuration="12.425919246s" totalImagesPullingTime="10.404751286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:50.425237614 +0000 UTC m=+13.824668887" watchObservedRunningTime="2026-10-29 22:13:50.425919246 +0000 UTC m=+13.825350518"
Oct 29 22:13:51.082998 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:51.082968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:51.083161 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:51.083087    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:13:51.083493 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:51.083472    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:51.083605 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:51.083574    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:13:52.082403 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:52.082233    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:52.082750 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:52.082479    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:13:52.324392 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:52.324285    2583 generic.go:356] "Generic (PLEG): container finished" podID="3611f6fc-6112-4b2e-8e22-75672c63c673" containerID="234b0b53b0e5ae9147a93313054f99ebc34dd84353019483d6bc55ca2c508539" exitCode=0
Oct 29 22:13:52.324392 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:52.324361    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xqtzn" event={"ID":"3611f6fc-6112-4b2e-8e22-75672c63c673","Type":"ContainerDied","Data":"234b0b53b0e5ae9147a93313054f99ebc34dd84353019483d6bc55ca2c508539"}
Oct 29 22:13:53.082956 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:53.082925    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:53.083424 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:53.083114    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:53.083424 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:53.083139    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:13:53.083424 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:53.083223    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:13:53.330382 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:53.330344    2583 generic.go:356] "Generic (PLEG): container finished" podID="3611f6fc-6112-4b2e-8e22-75672c63c673" containerID="80b6e52b371ff5e2060bc0104fd1aec8be6b4b99516b7e3129263218bbac7ca3" exitCode=0
Oct 29 22:13:53.330560 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:53.330415    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xqtzn" event={"ID":"3611f6fc-6112-4b2e-8e22-75672c63c673","Type":"ContainerDied","Data":"80b6e52b371ff5e2060bc0104fd1aec8be6b4b99516b7e3129263218bbac7ca3"}
Oct 29 22:13:54.082086 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:54.082056    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:54.082283 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:54.082165    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:13:54.335318 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:54.335252    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xqtzn" event={"ID":"3611f6fc-6112-4b2e-8e22-75672c63c673","Type":"ContainerStarted","Data":"ff693e9e3a6c2375ef33dc85dc7c2ef3d525016ee5d0e177f3d60c729ad934d2"}
Oct 29 22:13:54.399478 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:54.399437    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xqtzn" podStartSLOduration=5.7633996419999995 podStartE2EDuration="16.399425027s" totalImagesPullingTime="10.636025385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:54.387426213 +0000 UTC m=+17.786857482" watchObservedRunningTime="2026-10-29 22:13:54.399425027 +0000 UTC m=+17.798856295"
Oct 29 22:13:54.789279 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:54.789126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:54.789279 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:54.789264    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:54.789474 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:54.789288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:54.789474 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:54.789333    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret podName:7d56328b-9fa0-45cb-996f-65c1c1b8785c nodeName:}" failed. No retries permitted until 2026-10-29 22:14:10.789303836 +0000 UTC m=+34.188735097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret") pod "global-pull-secret-syncer-stxld" (UID: "7d56328b-9fa0-45cb-996f-65c1c1b8785c") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:13:54.789474 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:54.789379    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:54.789474 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:54.789420    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs podName:a97d96fb-da2b-4064-b6bb-ac7f3cf2270b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:10.789405244 +0000 UTC m=+34.188836490 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs") pod "network-metrics-daemon-l9rds" (UID: "a97d96fb-da2b-4064-b6bb-ac7f3cf2270b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:13:54.890416 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:54.890360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmzq\" (UniqueName: \"kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq\") pod \"network-check-target-sd9mh\" (UID: \"5345c4bc-ec8a-454b-83bf-363e0d1432e6\") " pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:54.890515 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:54.890496    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:13:54.890555 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:54.890516    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:13:54.890555 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:54.890525    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rrmzq for pod openshift-network-diagnostics/network-check-target-sd9mh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:54.890612 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:54.890562    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq podName:5345c4bc-ec8a-454b-83bf-363e0d1432e6 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:10.890551905 +0000 UTC m=+34.289983152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrmzq" (UniqueName: "kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq") pod "network-check-target-sd9mh" (UID: "5345c4bc-ec8a-454b-83bf-363e0d1432e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:13:55.083002 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:55.082975    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:55.083002 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:55.083003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:55.083487 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:55.083454    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:13:55.084150 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:55.084117    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:13:55.340258 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:55.340208    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-774ht" event={"ID":"eb9f947e-2e34-4b91-9e2b-1edafb918fef","Type":"ContainerStarted","Data":"4769ca3c290931b1db492d724acfe3cb14e7ee7ea7f70fbc5c9ca688b118da09"}
Oct 29 22:13:55.340561 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:55.340268    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-774ht" event={"ID":"eb9f947e-2e34-4b91-9e2b-1edafb918fef","Type":"ContainerStarted","Data":"273b22fa47eb45e2e2cfe9251f128e22ea44fa8c7483db64c198ab95976024dd"}
Oct 29 22:13:55.340561 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:55.340297    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-774ht" event={"ID":"eb9f947e-2e34-4b91-9e2b-1edafb918fef","Type":"ContainerStarted","Data":"74f134ebabf061155618a96f302bc3a05250faa22eb62cfc8da6627201592ff7"}
Oct 29 22:13:55.340561 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:55.340375    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-774ht" event={"ID":"eb9f947e-2e34-4b91-9e2b-1edafb918fef","Type":"ContainerStarted","Data":"e8e5b60c0e19a190733d68af41b273f6e76de5323d4569b3bf859128553b4af3"}
Oct 29 22:13:55.340561 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:55.340405    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-774ht" event={"ID":"eb9f947e-2e34-4b91-9e2b-1edafb918fef","Type":"ContainerStarted","Data":"7267d9c256d897f5893cccf2038c1688aa0dd916eceea3152d540c0b16707f99"}
Oct 29 22:13:55.340561 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:55.340435    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-774ht" event={"ID":"eb9f947e-2e34-4b91-9e2b-1edafb918fef","Type":"ContainerStarted","Data":"c2f765f9d134b0ee6946554fe1e9a1efb649544b1f1a017ea17309b49e6dc406"}
Oct 29 22:13:56.082597 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:56.082561    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:56.082738 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:56.082657    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:13:57.083378 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:57.083304    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:57.083775 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:57.083411    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:13:57.083775 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:57.083468    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:57.083775 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:57.083554    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:13:57.347709 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:57.347651    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-774ht" event={"ID":"eb9f947e-2e34-4b91-9e2b-1edafb918fef","Type":"ContainerStarted","Data":"7e6829064eddac31f62784f53f7acdf4fa0a39857d91f0aa9a6e996fe826a6cd"}
Oct 29 22:13:58.081975 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:58.081947    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:13:58.082119 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:58.082037    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:13:59.082341 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:59.082294    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:13:59.082965 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:59.082453    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:13:59.082965 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:59.082501    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:13:59.082965 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:13:59.082608    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:13:59.354634 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:59.354606    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-774ht" event={"ID":"eb9f947e-2e34-4b91-9e2b-1edafb918fef","Type":"ContainerStarted","Data":"dffa3600cdbcfda4b95b3998d3ac85aa9085871ae2336a62aface7442c688263"}
Oct 29 22:13:59.354849 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:59.354835    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:59.354913 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:59.354901    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:59.369217 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:59.369197    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:13:59.433185 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:13:59.433148    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-774ht" podStartSLOduration=6.346691717 podStartE2EDuration="21.433137271s" totalImagesPullingTime="15.086445554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:59.432943149 +0000 UTC m=+22.832374414" watchObservedRunningTime="2026-10-29 22:13:59.433137271 +0000 UTC m=+22.832568539"
Oct 29 22:14:00.082085 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:00.082053    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:00.082265 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:00.082150    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:14:00.357200 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:00.357128    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:00.371568 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:00.371548    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:00.923945 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:00.923915    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-stxld"]
Oct 29 22:14:00.924159 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:00.924044    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:00.924159 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:00.924117    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:14:00.926969 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:00.926942    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sd9mh"]
Oct 29 22:14:00.927089 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:00.927043    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:00.927152 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:00.927132    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:14:00.927632 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:00.927608    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l9rds"]
Oct 29 22:14:00.927738 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:00.927720    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:00.927971 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:00.927947    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:14:03.082961 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:03.082779    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:03.083361 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:03.082778    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:03.083361 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:03.083036    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:14:03.083361 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:03.082780    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:03.083361 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:03.083148    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:14:03.083361 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:03.083195    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:14:05.082157 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:05.082086    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:05.082157 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:05.082103    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:05.082663 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:05.082087    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:05.082663 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:05.082214    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:14:05.082663 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:05.082289    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:14:05.082663 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:05.082404    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:14:07.083582 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.083557    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:07.083938 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.083632    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-stxld" podUID="7d56328b-9fa0-45cb-996f-65c1c1b8785c"
Oct 29 22:14:07.083938 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.083752    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:07.083938 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.083883    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9rds" podUID="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b"
Oct 29 22:14:07.083938 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.083917    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:07.084106 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.083993    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sd9mh" podUID="5345c4bc-ec8a-454b-83bf-363e0d1432e6"
Oct 29 22:14:07.454558 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.454536    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeReady"
Oct 29 22:14:07.454670 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.454616    2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 29 22:14:07.500244 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.500220    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-stxg2"]
Oct 29 22:14:07.551807 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.551781    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2z8f7"]
Oct 29 22:14:07.551915 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.551892    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-stxg2"
Oct 29 22:14:07.565134 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.565112    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"]
Oct 29 22:14:07.565252 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.565237    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2z8f7"
Oct 29 22:14:07.567080 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.567063    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 29 22:14:07.567377 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.567357    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 29 22:14:07.576431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.576413    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6zmj5"]
Oct 29 22:14:07.576552 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.576533    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:07.585917 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.585898    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wfpts"]
Oct 29 22:14:07.586003 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.585989    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6zmj5"
Oct 29 22:14:07.598183 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.598146    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 29 22:14:07.598465 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.598437    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cwsdq\""
Oct 29 22:14:07.606424 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.606406    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f8746f6cf-j8g6f"]
Oct 29 22:14:07.606546 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.606530    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.630701 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.630683    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2"]
Oct 29 22:14:07.630817 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.630806    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.660532 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.660516    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"]
Oct 29 22:14:07.660629 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.660616    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2"
Oct 29 22:14:07.666974 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.666957    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 22:14:07.667195 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.667139    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 22:14:07.667195 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.667186    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:14:07.667352 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.667206    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 29 22:14:07.667352 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.667295    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:14:07.677906 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.677882    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7b6b64f55f-9l4xp"]
Oct 29 22:14:07.678005 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.677981    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:07.679739 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.679720    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 29 22:14:07.689914 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.689899    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 29 22:14:07.689914 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.689903    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 29 22:14:07.690485 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.690467    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"]
Oct 29 22:14:07.690602 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.690587    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.691496 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.691476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l65cw\" (UniqueName: \"kubernetes.io/projected/31bc5aff-808a-4d45-bcac-80d1bd626d47-kube-api-access-l65cw\") pod \"kas-connection-checker-685895c7c9-stxg2\" (UID: \"31bc5aff-808a-4d45-bcac-80d1bd626d47\") " pod="kube-system/kas-connection-checker-685895c7c9-stxg2"
Oct 29 22:14:07.691576 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.691513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb84513d-ca51-4920-bfc5-4240aa395064-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.691576 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.691532    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5pzj\" (UniqueName: \"kubernetes.io/projected/fb84513d-ca51-4920-bfc5-4240aa395064-kube-api-access-w5pzj\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.691642 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.691588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:07.691642 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.691635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52gp6\" (UniqueName: \"kubernetes.io/projected/ccb78139-7ccc-4ac8-87bf-264d048488c4-kube-api-access-52gp6\") pod \"kas-connection-checker-685895c7c9-2z8f7\" (UID: \"ccb78139-7ccc-4ac8-87bf-264d048488c4\") " pod="kube-system/kas-connection-checker-685895c7c9-2z8f7"
Oct 29 22:14:07.691718 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.691652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb84513d-ca51-4920-bfc5-4240aa395064-tmp\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.691718 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.691668    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb84513d-ca51-4920-bfc5-4240aa395064-snapshots\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.691718 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.691682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb84513d-ca51-4920-bfc5-4240aa395064-service-ca-bundle\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.691718 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.691696    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljh6n\" (UniqueName: \"kubernetes.io/projected/1e6952c3-e47b-4c03-8804-04de63d8f1ee-kube-api-access-ljh6n\") pod \"kas-connection-checker-685895c7c9-6zmj5\" (UID: \"1e6952c3-e47b-4c03-8804-04de63d8f1ee\") " pod="kube-system/kas-connection-checker-685895c7c9-6zmj5"
Oct 29 22:14:07.691837 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.691811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb84513d-ca51-4920-bfc5-4240aa395064-serving-cert\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.691881 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.691863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:07.691958 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.691942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r86gd\" (UniqueName: \"kubernetes.io/projected/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-kube-api-access-r86gd\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:07.698927 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.698902    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 29 22:14:07.699003 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.698936    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 29 22:14:07.699003 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.698946    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 29 22:14:07.699003 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.698963    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 29 22:14:07.699003 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.698972    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hkdgm\""
Oct 29 22:14:07.699147 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.699055    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w4b7h\""
Oct 29 22:14:07.699147 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.699070    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rzd6j\""
Oct 29 22:14:07.699205 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.699178    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 29 22:14:07.699205 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.699180    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x9frc\""
Oct 29 22:14:07.703874 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.703857    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 29 22:14:07.704078 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.704066    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 29 22:14:07.704365 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.704352    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 29 22:14:07.704420 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.704372    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n84hz\""
Oct 29 22:14:07.709773 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.709732    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"]
Oct 29 22:14:07.709845 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.709839    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.727422 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.727404    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"]
Oct 29 22:14:07.727526 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.727512    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:07.743450 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.743431    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-w9grl"]
Oct 29 22:14:07.743543 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.743530    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:07.755215 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.755199    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22"]
Oct 29 22:14:07.755304 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.755292    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:07.761602 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.761588    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9f64d5bbb-8pnfd"]
Oct 29 22:14:07.761702 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.761687    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22"
Oct 29 22:14:07.767883 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.767864    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 29 22:14:07.768031 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.767890    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:07.768031 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.767914    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 29 22:14:07.768031 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.767917    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:07.768031 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.767930    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:07.768031 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.767948    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:07.768031 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.768022    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:07.768689 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.768667    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:07.768893 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.768816    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 29 22:14:07.768893 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.768821    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:07.768893 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.768848    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 29 22:14:07.768893 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.768858    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 22:14:07.768893 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.768871    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:07.768893 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.768879    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:07.769137 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.768914    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 29 22:14:07.769137 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.768917    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 22:14:07.769137 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.768928    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:07.769238 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.769218    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 29 22:14:07.769641 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.769628    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 29 22:14:07.775124 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.775102    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:14:07.775124 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.775114    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:14:07.786451 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.786429    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5sn5b"]
Oct 29 22:14:07.786542 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.786528    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.792244 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792225    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb84513d-ca51-4920-bfc5-4240aa395064-snapshots\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.792351 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb84513d-ca51-4920-bfc5-4240aa395064-service-ca-bundle\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.792351 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792276    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljh6n\" (UniqueName: \"kubernetes.io/projected/1e6952c3-e47b-4c03-8804-04de63d8f1ee-kube-api-access-ljh6n\") pod \"kas-connection-checker-685895c7c9-6zmj5\" (UID: \"1e6952c3-e47b-4c03-8804-04de63d8f1ee\") " pod="kube-system/kas-connection-checker-685895c7c9-6zmj5"
Oct 29 22:14:07.792351 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792306    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-stats-auth\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.792351 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792341    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.792544 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792363    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-bound-sa-token\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.792544 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r86gd\" (UniqueName: \"kubernetes.io/projected/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-kube-api-access-r86gd\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:07.792544 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792452    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-default-certificate\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.792690 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792575    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:07.792690 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb84513d-ca51-4920-bfc5-4240aa395064-tmp\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.792690 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.792814 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5dd\" (UniqueName: \"kubernetes.io/projected/b822897d-559d-47b4-b9e8-9ba7f417a7bd-kube-api-access-8k5dd\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:07.792814 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7496f437-cb3c-4867-af37-16835f1134a1-trusted-ca\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.792885 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7496f437-cb3c-4867-af37-16835f1134a1-serving-cert\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.792885 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792831    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb84513d-ca51-4920-bfc5-4240aa395064-snapshots\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.792885 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb84513d-ca51-4920-bfc5-4240aa395064-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.792885 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792859    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5pzj\" (UniqueName: \"kubernetes.io/projected/fb84513d-ca51-4920-bfc5-4240aa395064-kube-api-access-w5pzj\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.793062 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkw58\" (UniqueName: \"kubernetes.io/projected/7496f437-cb3c-4867-af37-16835f1134a1-kube-api-access-bkw58\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.793062 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792895    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb84513d-ca51-4920-bfc5-4240aa395064-service-ca-bundle\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.793062 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7496f437-cb3c-4867-af37-16835f1134a1-config\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.793062 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792928    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb84513d-ca51-4920-bfc5-4240aa395064-tmp\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.793062 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.792935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb84513d-ca51-4920-bfc5-4240aa395064-serving-cert\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.793271 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:07.793271 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793170    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwxqn\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-kube-api-access-gwxqn\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.793271 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-image-registry-private-configuration\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.793271 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:07.793455 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.793455 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:07.793455 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793386    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplzf\" (UniqueName: \"kubernetes.io/projected/6f6f3621-6d10-4afb-9438-c73cc3d4b13d-kube-api-access-dplzf\") pod \"volume-data-source-validator-58c9f5f46d-vmtz2\" (UID: \"6f6f3621-6d10-4afb-9438-c73cc3d4b13d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2"
Oct 29 22:14:07.793603 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.793501    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:07.793603 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.793552    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls podName:f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d nodeName:}" failed. No retries permitted until 2026-10-29 22:14:08.293536043 +0000 UTC m=+31.692967292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n98ww" (UID: "f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d") : secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:07.793603 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-installation-pull-secrets\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.793719 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-trusted-ca\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.793719 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l65cw\" (UniqueName: \"kubernetes.io/projected/31bc5aff-808a-4d45-bcac-80d1bd626d47-kube-api-access-l65cw\") pod \"kas-connection-checker-685895c7c9-stxg2\" (UID: \"31bc5aff-808a-4d45-bcac-80d1bd626d47\") " pod="kube-system/kas-connection-checker-685895c7c9-stxg2"
Oct 29 22:14:07.793719 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-certificates\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.793834 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0da83403-18e0-4abd-8bbb-9323e80a3959-ca-trust-extracted\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.793834 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793736    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb84513d-ca51-4920-bfc5-4240aa395064-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.793914 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52gp6\" (UniqueName: \"kubernetes.io/projected/ccb78139-7ccc-4ac8-87bf-264d048488c4-kube-api-access-52gp6\") pod \"kas-connection-checker-685895c7c9-2z8f7\" (UID: \"ccb78139-7ccc-4ac8-87bf-264d048488c4\") " pod="kube-system/kas-connection-checker-685895c7c9-2z8f7"
Oct 29 22:14:07.793914 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972vc\" (UniqueName: \"kubernetes.io/projected/19e92d72-5f01-4c0f-9739-e094b74d5825-kube-api-access-972vc\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.794010 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.793948    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:07.796481 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.796462    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb84513d-ca51-4920-bfc5-4240aa395064-serving-cert\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.798853 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.798834    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 29 22:14:07.798853 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.798850    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 29 22:14:07.799047 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.799032    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-d5ps2\""
Oct 29 22:14:07.799100 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.799043    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r2ft9\""
Oct 29 22:14:07.799100 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.799072    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 29 22:14:07.799191 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.799114    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-s4dxs\""
Oct 29 22:14:07.799191 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.799165    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-k6pmr\""
Oct 29 22:14:07.799191 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.799172    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 29 22:14:07.799191 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.799158    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6sn8x\""
Oct 29 22:14:07.807335 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.807311    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mld7c"]
Oct 29 22:14:07.807408 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.807390    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:07.811449 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.811424    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljh6n\" (UniqueName: \"kubernetes.io/projected/1e6952c3-e47b-4c03-8804-04de63d8f1ee-kube-api-access-ljh6n\") pod \"kas-connection-checker-685895c7c9-6zmj5\" (UID: \"1e6952c3-e47b-4c03-8804-04de63d8f1ee\") " pod="kube-system/kas-connection-checker-685895c7c9-6zmj5"
Oct 29 22:14:07.811547 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.811531    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l65cw\" (UniqueName: \"kubernetes.io/projected/31bc5aff-808a-4d45-bcac-80d1bd626d47-kube-api-access-l65cw\") pod \"kas-connection-checker-685895c7c9-stxg2\" (UID: \"31bc5aff-808a-4d45-bcac-80d1bd626d47\") " pod="kube-system/kas-connection-checker-685895c7c9-stxg2"
Oct 29 22:14:07.811547 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.811540    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r86gd\" (UniqueName: \"kubernetes.io/projected/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-kube-api-access-r86gd\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:07.811772 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.811756    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5pzj\" (UniqueName: \"kubernetes.io/projected/fb84513d-ca51-4920-bfc5-4240aa395064-kube-api-access-w5pzj\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.811862 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.811847    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52gp6\" (UniqueName: \"kubernetes.io/projected/ccb78139-7ccc-4ac8-87bf-264d048488c4-kube-api-access-52gp6\") pod \"kas-connection-checker-685895c7c9-2z8f7\" (UID: \"ccb78139-7ccc-4ac8-87bf-264d048488c4\") " pod="kube-system/kas-connection-checker-685895c7c9-2z8f7"
Oct 29 22:14:07.817529 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817510    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-stxg2"]
Oct 29 22:14:07.817630 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817548    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2z8f7"]
Oct 29 22:14:07.817630 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817572    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6zmj5"]
Oct 29 22:14:07.817630 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817609    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2"]
Oct 29 22:14:07.817630 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817631    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"]
Oct 29 22:14:07.817768 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817652    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-w9grl"]
Oct 29 22:14:07.817768 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817659    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mld7c"
Oct 29 22:14:07.817768 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817672    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"]
Oct 29 22:14:07.817768 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817691    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22"]
Oct 29 22:14:07.817768 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817712    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"]
Oct 29 22:14:07.817768 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817729    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"]
Oct 29 22:14:07.817768 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817750    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f8746f6cf-j8g6f"]
Oct 29 22:14:07.817768 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817771    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wfpts"]
Oct 29 22:14:07.818051 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817796    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5sn5b"]
Oct 29 22:14:07.818051 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817822    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mld7c"]
Oct 29 22:14:07.818051 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817842    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b6b64f55f-9l4xp"]
Oct 29 22:14:07.818051 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817860    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9f64d5bbb-8pnfd"]
Oct 29 22:14:07.818051 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.817880    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"]
Oct 29 22:14:07.867710 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.867687    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:14:07.867793 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.867721    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:14:07.867793 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.867730    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:14:07.873924 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.873908    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-stxg2"
Oct 29 22:14:07.884163 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.884150    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2z8f7"
Oct 29 22:14:07.894487 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.894468    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-trusted-ca\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.894561 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.894501    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pkdx\" (UniqueName: \"kubernetes.io/projected/a342967b-704c-4947-92fa-e2dc475675a0-kube-api-access-6pkdx\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:07.894561 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.894529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxqn\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-kube-api-access-gwxqn\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.894640 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.894591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-image-registry-private-configuration\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.894640 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.894622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/20f39049-d7a1-46d8-8ebc-c0159906d689-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:07.894761 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.894745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.894811 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.894768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7099dc88-0238-49a5-b0fb-8d008e1b040e-tmp-dir\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:07.894811 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.894787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-image-registry-private-configuration\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.894922 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.894832    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.894970 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.894930    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:07.894970 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.894944    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8746f6cf-j8g6f: secret "image-registry-tls" not found
Oct 29 22:14:07.894970 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.894943    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a342967b-704c-4947-92fa-e2dc475675a0-serving-cert\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:07.895108 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.894988    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls podName:0da83403-18e0-4abd-8bbb-9323e80a3959 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:08.394971004 +0000 UTC m=+31.794402256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls") pod "image-registry-6f8746f6cf-j8g6f" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959") : secret "image-registry-tls" not found
Oct 29 22:14:07.895108 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f465b95-5af5-4d92-921b-24b0800934a0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:07.895108 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplzf\" (UniqueName: \"kubernetes.io/projected/6f6f3621-6d10-4afb-9438-c73cc3d4b13d-kube-api-access-dplzf\") pod \"volume-data-source-validator-58c9f5f46d-vmtz2\" (UID: \"6f6f3621-6d10-4afb-9438-c73cc3d4b13d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2"
Oct 29 22:14:07.895108 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-installation-pull-secrets\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.895108 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895095    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-trusted-ca\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.895369 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895135    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-certificates\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.895369 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7099dc88-0238-49a5-b0fb-8d008e1b040e-config-volume\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:07.895369 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg246\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-kube-api-access-tg246\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.895369 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0da83403-18e0-4abd-8bbb-9323e80a3959-ca-trust-extracted\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.895369 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972vc\" (UniqueName: \"kubernetes.io/projected/19e92d72-5f01-4c0f-9739-e094b74d5825-kube-api-access-972vc\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.895369 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-stats-auth\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.895369 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895345    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.895369 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-bound-sa-token\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57h7r\" (UniqueName: \"kubernetes.io/projected/9f465b95-5af5-4d92-921b-24b0800934a0-kube-api-access-57h7r\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-default-certificate\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895547    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a342967b-704c-4947-92fa-e2dc475675a0-config\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgwps\" (UniqueName: \"kubernetes.io/projected/61824bf7-eca8-4554-8c54-81153bdcca82-kube-api-access-tgwps\") pod \"network-check-source-6f78d59f99-j4c22\" (UID: \"61824bf7-eca8-4554-8c54-81153bdcca82\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895609    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-installation-pull-secrets\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5rt\" (UniqueName: \"kubernetes.io/projected/7099dc88-0238-49a5-b0fb-8d008e1b040e-kube-api-access-5d5rt\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895679    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-certificates\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5dd\" (UniqueName: \"kubernetes.io/projected/b822897d-559d-47b4-b9e8-9ba7f417a7bd-kube-api-access-8k5dd\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895741    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-certificates\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7496f437-cb3c-4867-af37-16835f1134a1-trusted-ca\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895786    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0f94ef8-7c40-437b-a27e-36583be8bfba-ca-trust-extracted\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.896946 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7496f437-cb3c-4867-af37-16835f1134a1-serving-cert\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkw58\" (UniqueName: \"kubernetes.io/projected/7496f437-cb3c-4867-af37-16835f1134a1-kube-api-access-bkw58\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-bound-sa-token\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7496f437-cb3c-4867-af37-16835f1134a1-config\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f465b95-5af5-4d92-921b-24b0800934a0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.895983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.896097    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.896141    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls podName:b822897d-559d-47b4-b9e8-9ba7f417a7bd nodeName:}" failed. No retries permitted until 2026-10-29 22:14:08.39612654 +0000 UTC m=+31.795557810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q9kr6" (UID: "b822897d-559d-47b4-b9e8-9ba7f417a7bd") : secret "samples-operator-tls" not found
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.896148    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-trusted-ca\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.896239    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.896286    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:08.396269875 +0000 UTC m=+31.795701135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : secret "router-metrics-certs-default" not found
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.896309    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:08.396297171 +0000 UTC m=+31.795728419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : configmap references non-existent config key: service-ca.crt
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.896559    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0da83403-18e0-4abd-8bbb-9323e80a3959-ca-trust-extracted\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.897362    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:07.897795 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.897461    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7496f437-cb3c-4867-af37-16835f1134a1-trusted-ca\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.898532 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.897720    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-image-registry-private-configuration\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.898532 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.898072    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7496f437-cb3c-4867-af37-16835f1134a1-config\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.898810 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.898732    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-stats-auth\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.898940 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.898922    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:14:07.899016 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.898994    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xsn6t\""
Oct 29 22:14:07.899067 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.899009    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:14:07.899067 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.899019    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2nqlz\""
Oct 29 22:14:07.899437 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.899418    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-installation-pull-secrets\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.900261 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.900241    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7496f437-cb3c-4867-af37-16835f1134a1-serving-cert\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.900757 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.900740    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-default-certificate\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.914364 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.914334    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxqn\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-kube-api-access-gwxqn\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.915061 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.915011    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-972vc\" (UniqueName: \"kubernetes.io/projected/19e92d72-5f01-4c0f-9739-e094b74d5825-kube-api-access-972vc\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:07.915170 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.915126    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5dd\" (UniqueName: \"kubernetes.io/projected/b822897d-559d-47b4-b9e8-9ba7f417a7bd-kube-api-access-8k5dd\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:07.916729 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.916710    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkw58\" (UniqueName: \"kubernetes.io/projected/7496f437-cb3c-4867-af37-16835f1134a1-kube-api-access-bkw58\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:07.916922 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.916906    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6zmj5"
Oct 29 22:14:07.918809 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.918790    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-bound-sa-token\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:07.920282 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.920253    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplzf\" (UniqueName: \"kubernetes.io/projected/6f6f3621-6d10-4afb-9438-c73cc3d4b13d-kube-api-access-dplzf\") pod \"volume-data-source-validator-58c9f5f46d-vmtz2\" (UID: \"6f6f3621-6d10-4afb-9438-c73cc3d4b13d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2"
Oct 29 22:14:07.936982 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.936962    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:07.986006 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.985772    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.996815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/20f39049-d7a1-46d8-8ebc-c0159906d689-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.996868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7099dc88-0238-49a5-b0fb-8d008e1b040e-tmp-dir\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.996902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-image-registry-private-configuration\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.996944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.996968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a342967b-704c-4947-92fa-e2dc475675a0-serving-cert\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.996993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f465b95-5af5-4d92-921b-24b0800934a0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.997028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7099dc88-0238-49a5-b0fb-8d008e1b040e-config-volume\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.997056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg246\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-kube-api-access-tg246\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.997134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57h7r\" (UniqueName: \"kubernetes.io/projected/9f465b95-5af5-4d92-921b-24b0800934a0-kube-api-access-57h7r\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.997207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.997238    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a342967b-704c-4947-92fa-e2dc475675a0-config\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.997267    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgwps\" (UniqueName: \"kubernetes.io/projected/61824bf7-eca8-4554-8c54-81153bdcca82-kube-api-access-tgwps\") pod \"network-check-source-6f78d59f99-j4c22\" (UID: \"61824bf7-eca8-4554-8c54-81153bdcca82\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.997294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-installation-pull-secrets\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.997338    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5rt\" (UniqueName: \"kubernetes.io/projected/7099dc88-0238-49a5-b0fb-8d008e1b040e-kube-api-access-5d5rt\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:07.998953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.997390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-certificates\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.997418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0f94ef8-7c40-437b-a27e-36583be8bfba-ca-trust-extracted\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.998345    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/20f39049-d7a1-46d8-8ebc-c0159906d689-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.998592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-bound-sa-token\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.998607    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7099dc88-0238-49a5-b0fb-8d008e1b040e-tmp-dir\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.998626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.998660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f465b95-5af5-4d92-921b-24b0800934a0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.998705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-trusted-ca\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.998734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pkdx\" (UniqueName: \"kubernetes.io/projected/a342967b-704c-4947-92fa-e2dc475675a0-kube-api-access-6pkdx\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.998768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/529f43fd-d894-45ac-9143-2dc520293e16-cert\") pod \"ingress-canary-mld7c\" (UID: \"529f43fd-d894-45ac-9143-2dc520293e16\") " pod="openshift-ingress-canary/ingress-canary-mld7c"
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.998794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vntmn\" (UniqueName: \"kubernetes.io/projected/529f43fd-d894-45ac-9143-2dc520293e16-kube-api-access-vntmn\") pod \"ingress-canary-mld7c\" (UID: \"529f43fd-d894-45ac-9143-2dc520293e16\") " pod="openshift-ingress-canary/ingress-canary-mld7c"
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.998953    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.999009    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert podName:20f39049-d7a1-46d8-8ebc-c0159906d689 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:08.498990653 +0000 UTC m=+31.898421920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9grl" (UID: "20f39049-d7a1-46d8-8ebc-c0159906d689") : secret "networking-console-plugin-cert" not found
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:07.999672    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a342967b-704c-4947-92fa-e2dc475675a0-config\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.999691    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.999711    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9f64d5bbb-8pnfd: secret "image-registry-tls" not found
Oct 29 22:14:07.999753 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:07.999766    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls podName:d0f94ef8-7c40-437b-a27e-36583be8bfba nodeName:}" failed. No retries permitted until 2026-10-29 22:14:08.49974687 +0000 UTC m=+31.899178119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls") pod "image-registry-9f64d5bbb-8pnfd" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba") : secret "image-registry-tls" not found
Oct 29 22:14:08.000536 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.000057    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:08.000536 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.000100    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls podName:7099dc88-0238-49a5-b0fb-8d008e1b040e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:08.500085893 +0000 UTC m=+31.899517159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls") pod "dns-default-5sn5b" (UID: "7099dc88-0238-49a5-b0fb-8d008e1b040e") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:08.002274 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.000826    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-certificates\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:08.002274 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.001071    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0f94ef8-7c40-437b-a27e-36583be8bfba-ca-trust-extracted\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:08.002274 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.001305    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f465b95-5af5-4d92-921b-24b0800934a0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:08.002274 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.001749    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-image-registry-private-configuration\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:08.002274 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.002204    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7099dc88-0238-49a5-b0fb-8d008e1b040e-config-volume\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:08.002274 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.002224    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-trusted-ca\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:08.002561 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.002416    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-installation-pull-secrets\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:08.006032 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.003623    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f465b95-5af5-4d92-921b-24b0800934a0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:08.007870 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.007750    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a342967b-704c-4947-92fa-e2dc475675a0-serving-cert\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:08.018646 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.016782    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg246\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-kube-api-access-tg246\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:08.022676 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.022533    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57h7r\" (UniqueName: \"kubernetes.io/projected/9f465b95-5af5-4d92-921b-24b0800934a0-kube-api-access-57h7r\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:08.023288 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.023246    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-bound-sa-token\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:08.024059 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.024039    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgwps\" (UniqueName: \"kubernetes.io/projected/61824bf7-eca8-4554-8c54-81153bdcca82-kube-api-access-tgwps\") pod \"network-check-source-6f78d59f99-j4c22\" (UID: \"61824bf7-eca8-4554-8c54-81153bdcca82\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22"
Oct 29 22:14:08.024654 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.024633    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pkdx\" (UniqueName: \"kubernetes.io/projected/a342967b-704c-4947-92fa-e2dc475675a0-kube-api-access-6pkdx\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:08.024737 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.024667    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5rt\" (UniqueName: \"kubernetes.io/projected/7099dc88-0238-49a5-b0fb-8d008e1b040e-kube-api-access-5d5rt\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:08.052633 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.052604    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:08.058007 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.057987    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:08.072882 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.072863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:08.099822 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.099802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/529f43fd-d894-45ac-9143-2dc520293e16-cert\") pod \"ingress-canary-mld7c\" (UID: \"529f43fd-d894-45ac-9143-2dc520293e16\") " pod="openshift-ingress-canary/ingress-canary-mld7c"
Oct 29 22:14:08.100159 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.099834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntmn\" (UniqueName: \"kubernetes.io/projected/529f43fd-d894-45ac-9143-2dc520293e16-kube-api-access-vntmn\") pod \"ingress-canary-mld7c\" (UID: \"529f43fd-d894-45ac-9143-2dc520293e16\") " pod="openshift-ingress-canary/ingress-canary-mld7c"
Oct 29 22:14:08.102043 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.102027    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/529f43fd-d894-45ac-9143-2dc520293e16-cert\") pod \"ingress-canary-mld7c\" (UID: \"529f43fd-d894-45ac-9143-2dc520293e16\") " pod="openshift-ingress-canary/ingress-canary-mld7c"
Oct 29 22:14:08.118082 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.118040    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntmn\" (UniqueName: \"kubernetes.io/projected/529f43fd-d894-45ac-9143-2dc520293e16-kube-api-access-vntmn\") pod \"ingress-canary-mld7c\" (UID: \"529f43fd-d894-45ac-9143-2dc520293e16\") " pod="openshift-ingress-canary/ingress-canary-mld7c"
Oct 29 22:14:08.118187 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.118161    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22"
Oct 29 22:14:08.181624 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.181600    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mld7c"
Oct 29 22:14:08.216369 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.216281    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-stxg2"]
Oct 29 22:14:08.218543 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.218516    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2z8f7"]
Oct 29 22:14:08.224114 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:14:08.224073    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bc5aff_808a_4d45_bcac_80d1bd626d47.slice/crio-a2423e0976b11dd60f5a69fcc2d098887e6bf9045380e7e8d71b6922462fea73 WatchSource:0}: Error finding container a2423e0976b11dd60f5a69fcc2d098887e6bf9045380e7e8d71b6922462fea73: Status 404 returned error can't find the container with id a2423e0976b11dd60f5a69fcc2d098887e6bf9045380e7e8d71b6922462fea73
Oct 29 22:14:08.272762 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.272632    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6zmj5"]
Oct 29 22:14:08.299842 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:14:08.299812    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6952c3_e47b_4c03_8804_04de63d8f1ee.slice/crio-9214ce184041408a36b09d343d6ed2817bef97abef2c4fbb12f635796b8481b6 WatchSource:0}: Error finding container 9214ce184041408a36b09d343d6ed2817bef97abef2c4fbb12f635796b8481b6: Status 404 returned error can't find the container with id 9214ce184041408a36b09d343d6ed2817bef97abef2c4fbb12f635796b8481b6
Oct 29 22:14:08.303431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.301760    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wfpts"]
Oct 29 22:14:08.303431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.302295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:08.303431 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.302508    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:08.303431 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.302574    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls podName:f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d nodeName:}" failed. No retries permitted until 2026-10-29 22:14:09.302549615 +0000 UTC m=+32.701980865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n98ww" (UID: "f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d") : secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:08.335920 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.335844    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2"]
Oct 29 22:14:08.341137 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:14:08.340587    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6f3621_6d10_4afb_9438_c73cc3d4b13d.slice/crio-2e8d034b86c896517d7333ff588c2fe0f1ec44b1d7660306d27d8802e3576605 WatchSource:0}: Error finding container 2e8d034b86c896517d7333ff588c2fe0f1ec44b1d7660306d27d8802e3576605: Status 404 returned error can't find the container with id 2e8d034b86c896517d7333ff588c2fe0f1ec44b1d7660306d27d8802e3576605
Oct 29 22:14:08.379472 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.379445    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"]
Oct 29 22:14:08.383725 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:14:08.383704    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7496f437_cb3c_4867_af37_16835f1134a1.slice/crio-6d4667d86df6e0ddc6785d2c7da6c7eac131b7c3f3d5f132c353d75d13ea9fee WatchSource:0}: Error finding container 6d4667d86df6e0ddc6785d2c7da6c7eac131b7c3f3d5f132c353d75d13ea9fee: Status 404 returned error can't find the container with id 6d4667d86df6e0ddc6785d2c7da6c7eac131b7c3f3d5f132c353d75d13ea9fee
Oct 29 22:14:08.386474 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.386446    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wfpts" event={"ID":"fb84513d-ca51-4920-bfc5-4240aa395064","Type":"ContainerStarted","Data":"1c8b2d1b8c54d8dacd521bba7922e4d71a65aad010fd7532554f3ead854fa929"}
Oct 29 22:14:08.388731 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.388662    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6zmj5" event={"ID":"1e6952c3-e47b-4c03-8804-04de63d8f1ee","Type":"ContainerStarted","Data":"9214ce184041408a36b09d343d6ed2817bef97abef2c4fbb12f635796b8481b6"}
Oct 29 22:14:08.390902 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.390857    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2z8f7" event={"ID":"ccb78139-7ccc-4ac8-87bf-264d048488c4","Type":"ContainerStarted","Data":"4a2437d7c6feb068ac8c90b800f394c221ef0f5c4815f2ad8481d0bb28b9d491"}
Oct 29 22:14:08.393378 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.393355    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-stxg2" event={"ID":"31bc5aff-808a-4d45-bcac-80d1bd626d47","Type":"ContainerStarted","Data":"a2423e0976b11dd60f5a69fcc2d098887e6bf9045380e7e8d71b6922462fea73"}
Oct 29 22:14:08.394892 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.394786    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2" event={"ID":"6f6f3621-6d10-4afb-9438-c73cc3d4b13d","Type":"ContainerStarted","Data":"2e8d034b86c896517d7333ff588c2fe0f1ec44b1d7660306d27d8802e3576605"}
Oct 29 22:14:08.401048 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.400862    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"]
Oct 29 22:14:08.403188 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.403168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:08.403272 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.403234    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:08.403272 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:14:08.403209    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f465b95_5af5_4d92_921b_24b0800934a0.slice/crio-c9de54c14081f90006f6c7c23ddf71f458c547b93db4173d1bdd04ab66e3cea0 WatchSource:0}: Error finding container c9de54c14081f90006f6c7c23ddf71f458c547b93db4173d1bdd04ab66e3cea0: Status 404 returned error can't find the container with id c9de54c14081f90006f6c7c23ddf71f458c547b93db4173d1bdd04ab66e3cea0
Oct 29 22:14:08.403272 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.403265    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 22:14:08.403435 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.403278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:08.403435 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.403334    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:09.403303688 +0000 UTC m=+32.802734935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : secret "router-metrics-certs-default" not found
Oct 29 22:14:08.403435 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.403361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:08.403435 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.403394    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 22:14:08.403558 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.403440    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls podName:b822897d-559d-47b4-b9e8-9ba7f417a7bd nodeName:}" failed. No retries permitted until 2026-10-29 22:14:09.403425811 +0000 UTC m=+32.802857096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q9kr6" (UID: "b822897d-559d-47b4-b9e8-9ba7f417a7bd") : secret "samples-operator-tls" not found
Oct 29 22:14:08.403558 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.403489    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:08.403558 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.403503    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8746f6cf-j8g6f: secret "image-registry-tls" not found
Oct 29 22:14:08.403558 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.403542    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls podName:0da83403-18e0-4abd-8bbb-9323e80a3959 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:09.40352958 +0000 UTC m=+32.802960850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls") pod "image-registry-6f8746f6cf-j8g6f" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959") : secret "image-registry-tls" not found
Oct 29 22:14:08.403693 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.403567    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:09.403556389 +0000 UTC m=+32.802987665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : configmap references non-existent config key: service-ca.crt
Oct 29 22:14:08.440591 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.440572    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"]
Oct 29 22:14:08.443163 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:14:08.443138    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda342967b_704c_4947_92fa_e2dc475675a0.slice/crio-802fc36f8fd0685143ad78ca1fe6300970f64a07df86cac38be2f0ad3c1bff9e WatchSource:0}: Error finding container 802fc36f8fd0685143ad78ca1fe6300970f64a07df86cac38be2f0ad3c1bff9e: Status 404 returned error can't find the container with id 802fc36f8fd0685143ad78ca1fe6300970f64a07df86cac38be2f0ad3c1bff9e
Oct 29 22:14:08.463098 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.463074    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22"]
Oct 29 22:14:08.464103 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:14:08.464084    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61824bf7_eca8_4554_8c54_81153bdcca82.slice/crio-cbbf4631a66f7278c8564c3a9b7f6872cc908ac3f8e8a78f82d6562e4b8d70a0 WatchSource:0}: Error finding container cbbf4631a66f7278c8564c3a9b7f6872cc908ac3f8e8a78f82d6562e4b8d70a0: Status 404 returned error can't find the container with id cbbf4631a66f7278c8564c3a9b7f6872cc908ac3f8e8a78f82d6562e4b8d70a0
Oct 29 22:14:08.503998 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.503976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:08.504084 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.504043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:08.504084 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.504081    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:08.504189 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.504104    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:08.504189 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.504119    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9f64d5bbb-8pnfd: secret "image-registry-tls" not found
Oct 29 22:14:08.504189 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.504158    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls podName:d0f94ef8-7c40-437b-a27e-36583be8bfba nodeName:}" failed. No retries permitted until 2026-10-29 22:14:09.504145441 +0000 UTC m=+32.903576691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls") pod "image-registry-9f64d5bbb-8pnfd" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba") : secret "image-registry-tls" not found
Oct 29 22:14:08.504189 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.504184    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:08.504391 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.504197    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 22:14:08.504391 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.504216    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls podName:7099dc88-0238-49a5-b0fb-8d008e1b040e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:09.504206757 +0000 UTC m=+32.903638029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls") pod "dns-default-5sn5b" (UID: "7099dc88-0238-49a5-b0fb-8d008e1b040e") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:08.504391 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:08.504249    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert podName:20f39049-d7a1-46d8-8ebc-c0159906d689 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:09.504233531 +0000 UTC m=+32.903664780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9grl" (UID: "20f39049-d7a1-46d8-8ebc-c0159906d689") : secret "networking-console-plugin-cert" not found
Oct 29 22:14:08.510315 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:08.510296    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mld7c"]
Oct 29 22:14:08.512338 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:14:08.512309    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529f43fd_d894_45ac_9143_2dc520293e16.slice/crio-90b12b994e0a8646661226fd42f6773cb7cd595b40ea72558fed477065377385 WatchSource:0}: Error finding container 90b12b994e0a8646661226fd42f6773cb7cd595b40ea72558fed477065377385: Status 404 returned error can't find the container with id 90b12b994e0a8646661226fd42f6773cb7cd595b40ea72558fed477065377385
Oct 29 22:14:09.088988 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.088960    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:09.089147 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.089004    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:09.091460 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.089447    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:09.103719 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.103594    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v7npq\""
Oct 29 22:14:09.103719 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.103638    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-977wj\""
Oct 29 22:14:09.103719 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.103645    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:14:09.103719 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.103691    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:14:09.104241 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.103731    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7vbxc\""
Oct 29 22:14:09.312172 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.312136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:09.312486 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.312438    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:09.312583 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.312571    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls podName:f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d nodeName:}" failed. No retries permitted until 2026-10-29 22:14:11.312551125 +0000 UTC m=+34.711982379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n98ww" (UID: "f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d") : secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:09.401637 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.401518    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mld7c" event={"ID":"529f43fd-d894-45ac-9143-2dc520293e16","Type":"ContainerStarted","Data":"90b12b994e0a8646661226fd42f6773cb7cd595b40ea72558fed477065377385"}
Oct 29 22:14:09.405000 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.404939    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg" event={"ID":"9f465b95-5af5-4d92-921b-24b0800934a0","Type":"ContainerStarted","Data":"c9de54c14081f90006f6c7c23ddf71f458c547b93db4173d1bdd04ab66e3cea0"}
Oct 29 22:14:09.408979 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.408916    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6" event={"ID":"a342967b-704c-4947-92fa-e2dc475675a0","Type":"ContainerStarted","Data":"802fc36f8fd0685143ad78ca1fe6300970f64a07df86cac38be2f0ad3c1bff9e"}
Oct 29 22:14:09.412955 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.412927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:09.413046 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.413010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:09.413110 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.413070    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:09.413160 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.413113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:09.413376 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.413283    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:09.413376 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.413299    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8746f6cf-j8g6f: secret "image-registry-tls" not found
Oct 29 22:14:09.413376 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.413320    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:11.413300148 +0000 UTC m=+34.812731416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : configmap references non-existent config key: service-ca.crt
Oct 29 22:14:09.413376 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.413355    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls podName:0da83403-18e0-4abd-8bbb-9323e80a3959 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:11.413345848 +0000 UTC m=+34.812777095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls") pod "image-registry-6f8746f6cf-j8g6f" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959") : secret "image-registry-tls" not found
Oct 29 22:14:09.413541 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.413389    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 22:14:09.413541 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.413445    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls podName:b822897d-559d-47b4-b9e8-9ba7f417a7bd nodeName:}" failed. No retries permitted until 2026-10-29 22:14:11.413428869 +0000 UTC m=+34.812860120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q9kr6" (UID: "b822897d-559d-47b4-b9e8-9ba7f417a7bd") : secret "samples-operator-tls" not found
Oct 29 22:14:09.413541 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.413448    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 22:14:09.413541 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.413502    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:11.413484881 +0000 UTC m=+34.812916148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : secret "router-metrics-certs-default" not found
Oct 29 22:14:09.415700 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.415120    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6zmj5" event={"ID":"1e6952c3-e47b-4c03-8804-04de63d8f1ee","Type":"ContainerStarted","Data":"c9576c58314ecd0511e1c9066c67c76b7773050a8d09595e8859fd5614867fa1"}
Oct 29 22:14:09.420072 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.420038    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22" event={"ID":"61824bf7-eca8-4554-8c54-81153bdcca82","Type":"ContainerStarted","Data":"cbbf4631a66f7278c8564c3a9b7f6872cc908ac3f8e8a78f82d6562e4b8d70a0"}
Oct 29 22:14:09.423623 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.423598    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2z8f7" event={"ID":"ccb78139-7ccc-4ac8-87bf-264d048488c4","Type":"ContainerStarted","Data":"2092c511819e7ef936819d1c9d93a8c713bf689d58f68fe4614517697f0c2576"}
Oct 29 22:14:09.428124 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.428100    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-stxg2" event={"ID":"31bc5aff-808a-4d45-bcac-80d1bd626d47","Type":"ContainerStarted","Data":"433a0c8e999c549569a48a80dad2f2ebb6b9ed214193f2b0338fa131dab4c8f4"}
Oct 29 22:14:09.432580 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.432556    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p" event={"ID":"7496f437-cb3c-4867-af37-16835f1134a1","Type":"ContainerStarted","Data":"6d4667d86df6e0ddc6785d2c7da6c7eac131b7c3f3d5f132c353d75d13ea9fee"}
Oct 29 22:14:09.452598 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.451584    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6zmj5" podStartSLOduration=335.451570485 podStartE2EDuration="5m35.451570485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:09.450630544 +0000 UTC m=+32.850061814" watchObservedRunningTime="2026-10-29 22:14:09.451570485 +0000 UTC m=+32.851001755"
Oct 29 22:14:09.488359 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.488197    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-2z8f7" podStartSLOduration=335.488182529 podStartE2EDuration="5m35.488182529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:09.487733899 +0000 UTC m=+32.887165169" watchObservedRunningTime="2026-10-29 22:14:09.488182529 +0000 UTC m=+32.887613799"
Oct 29 22:14:09.508164 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.508129    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:09.516870 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.514168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:09.516870 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.514313    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:09.516870 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.514399    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:09.516870 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.514537    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:09.516870 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.514588    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls podName:7099dc88-0238-49a5-b0fb-8d008e1b040e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:11.514573627 +0000 UTC m=+34.914004887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls") pod "dns-default-5sn5b" (UID: "7099dc88-0238-49a5-b0fb-8d008e1b040e") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:09.516870 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.514900    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:09.516870 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.514932    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9f64d5bbb-8pnfd: secret "image-registry-tls" not found
Oct 29 22:14:09.516870 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.514975    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls podName:d0f94ef8-7c40-437b-a27e-36583be8bfba nodeName:}" failed. No retries permitted until 2026-10-29 22:14:11.514961769 +0000 UTC m=+34.914393033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls") pod "image-registry-9f64d5bbb-8pnfd" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba") : secret "image-registry-tls" not found
Oct 29 22:14:09.516870 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.516766    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 22:14:09.516870 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:09.516811    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert podName:20f39049-d7a1-46d8-8ebc-c0159906d689 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:11.51679649 +0000 UTC m=+34.916227750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9grl" (UID: "20f39049-d7a1-46d8-8ebc-c0159906d689") : secret "networking-console-plugin-cert" not found
Oct 29 22:14:09.525533 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.525493    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-stxg2" podStartSLOduration=335.525478753 podStartE2EDuration="5m35.525478753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:09.521994025 +0000 UTC m=+32.921425293" watchObservedRunningTime="2026-10-29 22:14:09.525478753 +0000 UTC m=+32.924910024"
Oct 29 22:14:09.672906 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:09.672786    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:10.091383 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.091305    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:10.445573 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.445530    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2" event={"ID":"6f6f3621-6d10-4afb-9438-c73cc3d4b13d","Type":"ContainerStarted","Data":"55beb3da67389f18c6a13e3598c82f9dce2ed9eadefb77ae971d25a31fa80c35"}
Oct 29 22:14:10.481991 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.481941    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2" podStartSLOduration=306.586618347 podStartE2EDuration="5m8.481922199s" totalImagesPullingTime="1.895303852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:10.481290096 +0000 UTC m=+33.880721361" watchObservedRunningTime="2026-10-29 22:14:10.481922199 +0000 UTC m=+33.881353471"
Oct 29 22:14:10.597058 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.596971    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:10.713868 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.713800    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:10.830608 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.830575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:10.830763 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.830633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:10.831249 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:10.831221    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 22:14:10.831368 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:10.831291    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs podName:a97d96fb-da2b-4064-b6bb-ac7f3cf2270b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:42.831276897 +0000 UTC m=+66.230708148 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs") pod "network-metrics-daemon-l9rds" (UID: "a97d96fb-da2b-4064-b6bb-ac7f3cf2270b") : secret "metrics-daemon-secret" not found
Oct 29 22:14:10.833494 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.833474    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:10.931286 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.931258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmzq\" (UniqueName: \"kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq\") pod \"network-check-target-sd9mh\" (UID: \"5345c4bc-ec8a-454b-83bf-363e0d1432e6\") " pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:10.936797 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.936743    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmzq\" (UniqueName: \"kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq\") pod \"network-check-target-sd9mh\" (UID: \"5345c4bc-ec8a-454b-83bf-363e0d1432e6\") " pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:10.955521 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.955495    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:10.967595 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.967570    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:10.977978 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:10.977914    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:11.101030 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.100994    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:11.281438 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.278199    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sd9mh"]
Oct 29 22:14:11.306034 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.306009    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-stxld"]
Oct 29 22:14:11.309361 ip-10-0-128-31 kubenswrapper[2583]: W1029 22:14:11.309332    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d56328b_9fa0_45cb_996f_65c1c1b8785c.slice/crio-30b6f54569da4ddd21c89d61a36d340e2d809a1325ae145988564b22ea17fc88 WatchSource:0}: Error finding container 30b6f54569da4ddd21c89d61a36d340e2d809a1325ae145988564b22ea17fc88: Status 404 returned error can't find the container with id 30b6f54569da4ddd21c89d61a36d340e2d809a1325ae145988564b22ea17fc88
Oct 29 22:14:11.335611 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.335092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:11.335611 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.335230    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:11.335611 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.335289    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls podName:f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d nodeName:}" failed. No retries permitted until 2026-10-29 22:14:15.335270142 +0000 UTC m=+38.734701390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n98ww" (UID: "f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d") : secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:11.437252 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.436202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:11.437252 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.436288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:11.437252 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.436365    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:11.437252 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.436399    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:11.437252 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.436543    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:11.437252 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.436559    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8746f6cf-j8g6f: secret "image-registry-tls" not found
Oct 29 22:14:11.437252 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.436611    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls podName:0da83403-18e0-4abd-8bbb-9323e80a3959 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:15.436594827 +0000 UTC m=+38.836026078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls") pod "image-registry-6f8746f6cf-j8g6f" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959") : secret "image-registry-tls" not found
Oct 29 22:14:11.437252 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.436982    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 22:14:11.437252 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.437037    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:15.437021167 +0000 UTC m=+38.836452418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : secret "router-metrics-certs-default" not found
Oct 29 22:14:11.437252 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.437119    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:15.437108377 +0000 UTC m=+38.836539627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : configmap references non-existent config key: service-ca.crt
Oct 29 22:14:11.437252 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.437177    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 22:14:11.437252 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.437203    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls podName:b822897d-559d-47b4-b9e8-9ba7f417a7bd nodeName:}" failed. No retries permitted until 2026-10-29 22:14:15.437196662 +0000 UTC m=+38.836627909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q9kr6" (UID: "b822897d-559d-47b4-b9e8-9ba7f417a7bd") : secret "samples-operator-tls" not found
Oct 29 22:14:11.448998 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.448966    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-stxld" event={"ID":"7d56328b-9fa0-45cb-996f-65c1c1b8785c","Type":"ContainerStarted","Data":"30b6f54569da4ddd21c89d61a36d340e2d809a1325ae145988564b22ea17fc88"}
Oct 29 22:14:11.451468 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.451431    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sd9mh" event={"ID":"5345c4bc-ec8a-454b-83bf-363e0d1432e6","Type":"ContainerStarted","Data":"f898c7635d1499a52e6eaa271ab9c34e5cbe954e7499ea2de69e97d38096b4ad"}
Oct 29 22:14:11.537261 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.537196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:11.537406 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.537262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:11.537406 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.537312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:11.537406 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.537386    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:11.537582 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.537411    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 22:14:11.537582 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.537454    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls podName:7099dc88-0238-49a5-b0fb-8d008e1b040e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:15.537434769 +0000 UTC m=+38.936866034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls") pod "dns-default-5sn5b" (UID: "7099dc88-0238-49a5-b0fb-8d008e1b040e") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:11.537582 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.537474    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:11.537582 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.537476    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert podName:20f39049-d7a1-46d8-8ebc-c0159906d689 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:15.537464304 +0000 UTC m=+38.936895555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9grl" (UID: "20f39049-d7a1-46d8-8ebc-c0159906d689") : secret "networking-console-plugin-cert" not found
Oct 29 22:14:11.537582 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.537482    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9f64d5bbb-8pnfd: secret "image-registry-tls" not found
Oct 29 22:14:11.537582 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:11.537507    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls podName:d0f94ef8-7c40-437b-a27e-36583be8bfba nodeName:}" failed. No retries permitted until 2026-10-29 22:14:15.537497672 +0000 UTC m=+38.936928918 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls") pod "image-registry-9f64d5bbb-8pnfd" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba") : secret "image-registry-tls" not found
Oct 29 22:14:11.941431 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:11.941397    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:12.047623 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:12.047588    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:13.476682 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:13.476616    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg" event={"ID":"9f465b95-5af5-4d92-921b-24b0800934a0","Type":"ContainerStarted","Data":"c715ed8c9ef4252f38eebce796362d32340fb1610083d5c998624563f8507041"}
Oct 29 22:14:13.479308 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:13.478933    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wfpts" event={"ID":"fb84513d-ca51-4920-bfc5-4240aa395064","Type":"ContainerStarted","Data":"2f61f374adcab3f31219b2e24b52ad05ace83d28684e07e6d9dba26cdfff28d0"}
Oct 29 22:14:13.517151 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:13.517095    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-wfpts" podStartSLOduration=306.490413484 podStartE2EDuration="5m11.517081916s" totalImagesPullingTime="5.026668432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:13.516079761 +0000 UTC m=+36.915511030" watchObservedRunningTime="2026-10-29 22:14:13.517081916 +0000 UTC m=+36.916513185"
Oct 29 22:14:14.181019 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.179547    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pd9qn"]
Oct 29 22:14:14.200474 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.200445    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.205305 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.205278    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pd9qn"]
Oct 29 22:14:14.219707 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.219686    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:14:14.219813 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.219790    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:14:14.231081 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.231059    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xpwxr\""
Oct 29 22:14:14.286779 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.286756    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:14.366147 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.366118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.366341 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.366233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbdg5\" (UniqueName: \"kubernetes.io/projected/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-api-access-hbdg5\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.366341 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.366296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9ac5e7-84bd-4b61-958b-f78cd316e348-crio-socket\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.366471 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.366341    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.366471 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.366388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9ac5e7-84bd-4b61-958b-f78cd316e348-data-volume\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.399553 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.399524    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:14.474438 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.472929    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbdg5\" (UniqueName: \"kubernetes.io/projected/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-api-access-hbdg5\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.474438 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.473005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9ac5e7-84bd-4b61-958b-f78cd316e348-crio-socket\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.474438 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.473029    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.474438 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.473070    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9ac5e7-84bd-4b61-958b-f78cd316e348-data-volume\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.474438 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.473161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.474438 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:14.473308    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:14.474438 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:14.473382    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls podName:df9ac5e7-84bd-4b61-958b-f78cd316e348 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.973365051 +0000 UTC m=+38.372796313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd9qn" (UID: "df9ac5e7-84bd-4b61-958b-f78cd316e348") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:14.474438 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.473798    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9ac5e7-84bd-4b61-958b-f78cd316e348-crio-socket\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.474438 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:14.473858    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 22:14:14.474438 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:14.473897    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm podName:df9ac5e7-84bd-4b61-958b-f78cd316e348 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:14.973883891 +0000 UTC m=+38.373315157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pd9qn" (UID: "df9ac5e7-84bd-4b61-958b-f78cd316e348") : configmap "kube-rbac-proxy" not found
Oct 29 22:14:14.474438 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.474192    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9ac5e7-84bd-4b61-958b-f78cd316e348-data-volume\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.498205 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.498179    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/0.log"
Oct 29 22:14:14.498709 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.498687    2583 generic.go:356] "Generic (PLEG): container finished" podID="7496f437-cb3c-4867-af37-16835f1134a1" containerID="c1d0f28968ae184a53413cd289652b1c6b1ffc1f56b1d3e2125e793a2017b26f" exitCode=255
Oct 29 22:14:14.499248 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.499218    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p" event={"ID":"7496f437-cb3c-4867-af37-16835f1134a1","Type":"ContainerDied","Data":"c1d0f28968ae184a53413cd289652b1c6b1ffc1f56b1d3e2125e793a2017b26f"}
Oct 29 22:14:14.499711 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.499691    2583 scope.go:122] "RemoveContainer" containerID="c1d0f28968ae184a53413cd289652b1c6b1ffc1f56b1d3e2125e793a2017b26f"
Oct 29 22:14:14.502837 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.502810    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbdg5\" (UniqueName: \"kubernetes.io/projected/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-api-access-hbdg5\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.512270 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.510803    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mld7c" event={"ID":"529f43fd-d894-45ac-9143-2dc520293e16","Type":"ContainerStarted","Data":"0ff863357a61be5234c43194b0451352ba075c6d1e7f035f22f5fbea4c235f96"}
Oct 29 22:14:14.518787 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.518709    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6" event={"ID":"a342967b-704c-4947-92fa-e2dc475675a0","Type":"ContainerStarted","Data":"4cb9f8be95497267181303b5dd9c87256d46723aac4132c8d7c9cfb4c1ff5bee"}
Oct 29 22:14:14.533379 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.532387    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:14.582557 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.582496    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mld7c" podStartSLOduration=1.84950429 podStartE2EDuration="7.582476635s" totalImagesPullingTime="5.732972345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:14.576956604 +0000 UTC m=+37.976387924" watchObservedRunningTime="2026-10-29 22:14:14.582476635 +0000 UTC m=+37.981907908"
Oct 29 22:14:14.615076 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.615022    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6" podStartSLOduration=307.288576788 podStartE2EDuration="5m12.615000639s" totalImagesPullingTime="5.326423851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:14.613655099 +0000 UTC m=+38.013086368" watchObservedRunningTime="2026-10-29 22:14:14.615000639 +0000 UTC m=+38.014431910"
Oct 29 22:14:14.652516 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.652480    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:14.764774 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.764708    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:14.977930 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.977902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.978063 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:14.978042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:14.978063 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:14.978054    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:14.978192 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:14.978119    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls podName:df9ac5e7-84bd-4b61-958b-f78cd316e348 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:15.978098918 +0000 UTC m=+39.377530168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd9qn" (UID: "df9ac5e7-84bd-4b61-958b-f78cd316e348") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:14.978192 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:14.978160    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 22:14:14.978259 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:14.978211    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm podName:df9ac5e7-84bd-4b61-958b-f78cd316e348 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:15.978198515 +0000 UTC m=+39.377629777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pd9qn" (UID: "df9ac5e7-84bd-4b61-958b-f78cd316e348") : configmap "kube-rbac-proxy" not found
Oct 29 22:14:15.044521 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.044499    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:15.166871 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.166844    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:15.301587 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.301528    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:15.381155 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.381127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:15.381362 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.381318    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:15.381451 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.381411    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls podName:f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d nodeName:}" failed. No retries permitted until 2026-10-29 22:14:23.381395025 +0000 UTC m=+46.780826290 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n98ww" (UID: "f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d") : secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:15.481771 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.481741    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:15.481885 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.481794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:15.481885 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.481871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:15.481981 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.481904    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 22:14:15.481981 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.481926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:15.481981 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.481957    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:15.481981 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.481967    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls podName:b822897d-559d-47b4-b9e8-9ba7f417a7bd nodeName:}" failed. No retries permitted until 2026-10-29 22:14:23.481948079 +0000 UTC m=+46.881379342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q9kr6" (UID: "b822897d-559d-47b4-b9e8-9ba7f417a7bd") : secret "samples-operator-tls" not found
Oct 29 22:14:15.481981 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.481971    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8746f6cf-j8g6f: secret "image-registry-tls" not found
Oct 29 22:14:15.482138 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.482016    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls podName:0da83403-18e0-4abd-8bbb-9323e80a3959 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:23.482001641 +0000 UTC m=+46.881432898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls") pod "image-registry-6f8746f6cf-j8g6f" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959") : secret "image-registry-tls" not found
Oct 29 22:14:15.482138 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.482034    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:23.48202422 +0000 UTC m=+46.881455473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : configmap references non-existent config key: service-ca.crt
Oct 29 22:14:15.482138 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.482064    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 22:14:15.482138 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.482095    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:23.482083399 +0000 UTC m=+46.881514664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : secret "router-metrics-certs-default" not found
Oct 29 22:14:15.522746 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.522715    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22" event={"ID":"61824bf7-eca8-4554-8c54-81153bdcca82","Type":"ContainerStarted","Data":"d1aa3e10527b615e514e8a55d163286063a653eee114a36cb9ce7e5669372a77"}
Oct 29 22:14:15.524486 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.524456    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sd9mh" event={"ID":"5345c4bc-ec8a-454b-83bf-363e0d1432e6","Type":"ContainerStarted","Data":"0bd52ed6b3a54e710a9ee216df5d50c64ff5711759b55e0394b9ca63b23cd1d5"}
Oct 29 22:14:15.524599 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.524563    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:15.526460 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.526443    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/1.log"
Oct 29 22:14:15.526931 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.526910    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/0.log"
Oct 29 22:14:15.527024 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.526956    2583 generic.go:356] "Generic (PLEG): container finished" podID="7496f437-cb3c-4867-af37-16835f1134a1" containerID="7c727fc6b491e3c7a5d226c6cccc83981ca0988ffe7eb5d016e5d71adef13b31" exitCode=255
Oct 29 22:14:15.527155 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.527133    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p" event={"ID":"7496f437-cb3c-4867-af37-16835f1134a1","Type":"ContainerDied","Data":"7c727fc6b491e3c7a5d226c6cccc83981ca0988ffe7eb5d016e5d71adef13b31"}
Oct 29 22:14:15.527251 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.527181    2583 scope.go:122] "RemoveContainer" containerID="c1d0f28968ae184a53413cd289652b1c6b1ffc1f56b1d3e2125e793a2017b26f"
Oct 29 22:14:15.527786 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.527567    2583 scope.go:122] "RemoveContainer" containerID="7c727fc6b491e3c7a5d226c6cccc83981ca0988ffe7eb5d016e5d71adef13b31"
Oct 29 22:14:15.527786 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.527735    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7gx9p_openshift-console-operator(7496f437-cb3c-4867-af37-16835f1134a1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p" podUID="7496f437-cb3c-4867-af37-16835f1134a1"
Oct 29 22:14:15.558106 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.558042    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg" podStartSLOduration=308.606968416 podStartE2EDuration="5m13.558031331s" totalImagesPullingTime="4.951062915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:14.65634611 +0000 UTC m=+38.055777381" watchObservedRunningTime="2026-10-29 22:14:15.558031331 +0000 UTC m=+38.957462644"
Oct 29 22:14:15.558684 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.558656    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22" podStartSLOduration=306.046412266 podStartE2EDuration="5m12.558650183s" totalImagesPullingTime="6.512237917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:15.557723795 +0000 UTC m=+38.957155075" watchObservedRunningTime="2026-10-29 22:14:15.558650183 +0000 UTC m=+38.958081452"
Oct 29 22:14:15.582537 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.582514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:15.582741 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.582727    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:15.583127 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.582788    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls podName:7099dc88-0238-49a5-b0fb-8d008e1b040e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:23.582773315 +0000 UTC m=+46.982204583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls") pod "dns-default-5sn5b" (UID: "7099dc88-0238-49a5-b0fb-8d008e1b040e") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:15.583127 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.582877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:15.583127 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.583020    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:15.583127 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.583038    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9f64d5bbb-8pnfd: secret "image-registry-tls" not found
Oct 29 22:14:15.583127 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.583040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:15.583127 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.583093    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls podName:d0f94ef8-7c40-437b-a27e-36583be8bfba nodeName:}" failed. No retries permitted until 2026-10-29 22:14:23.583078642 +0000 UTC m=+46.982509896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls") pod "image-registry-9f64d5bbb-8pnfd" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba") : secret "image-registry-tls" not found
Oct 29 22:14:15.583127 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.583113    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 22:14:15.583526 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.583186    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert podName:20f39049-d7a1-46d8-8ebc-c0159906d689 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:23.583175888 +0000 UTC m=+46.982607141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9grl" (UID: "20f39049-d7a1-46d8-8ebc-c0159906d689") : secret "networking-console-plugin-cert" not found
Oct 29 22:14:15.631098 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.631037    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sd9mh" podStartSLOduration=33.796213111 podStartE2EDuration="37.631022935s" totalImagesPullingTime="3.834809824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:15.629252659 +0000 UTC m=+39.028683927" watchObservedRunningTime="2026-10-29 22:14:15.631022935 +0000 UTC m=+39.030454196"
Oct 29 22:14:15.774822 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.774791    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:15.986953 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.986906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:15.987199 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:15.987042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:15.987199 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.987048    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 22:14:15.987199 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.987192    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm podName:df9ac5e7-84bd-4b61-958b-f78cd316e348 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:17.987177381 +0000 UTC m=+41.386608642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pd9qn" (UID: "df9ac5e7-84bd-4b61-958b-f78cd316e348") : configmap "kube-rbac-proxy" not found
Oct 29 22:14:15.987199 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.987123    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:15.987441 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:15.987227    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls podName:df9ac5e7-84bd-4b61-958b-f78cd316e348 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:17.987221006 +0000 UTC m=+41.386652267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd9qn" (UID: "df9ac5e7-84bd-4b61-958b-f78cd316e348") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:16.375730 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:16.375651    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:16.532727 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:16.532702    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/1.log"
Oct 29 22:14:16.533195 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:16.533094    2583 scope.go:122] "RemoveContainer" containerID="7c727fc6b491e3c7a5d226c6cccc83981ca0988ffe7eb5d016e5d71adef13b31"
Oct 29 22:14:16.533318 ip-10-0-128-31 kubenswrapper[2583]: E1029 22:14:16.533295    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7gx9p_openshift-console-operator(7496f437-cb3c-4867-af37-16835f1134a1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p" podUID="7496f437-cb3c-4867-af37-16835f1134a1"
Oct 29 22:14:17.911657 ip-10-0-128-31 kubenswrapper[2583]: I1029 22:14:17.911567    2583 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:14:17.911751 ip-10-0-128-31 systemd[1]: Stopping Kubernetes Kubelet...
Oct 29 22:14:17.929690 ip-10-0-128-31 systemd[1]: kubelet.service: Deactivated successfully.
Oct 29 22:14:17.929850 ip-10-0-128-31 systemd[1]: Stopped Kubernetes Kubelet.
Oct 29 22:14:17.929896 ip-10-0-128-31 systemd[1]: kubelet.service: Consumed 2.834s CPU time.
Oct 29 22:14:17.945968 ip-10-0-128-31 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:14:18.031287 ip-10-0-128-31 kubenswrapper[4860]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:14:18.031287 ip-10-0-128-31 kubenswrapper[4860]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:14:18.031287 ip-10-0-128-31 kubenswrapper[4860]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:14:18.031287 ip-10-0-128-31 kubenswrapper[4860]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:14:18.033623 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033547    4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:18.033623 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033624    4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033628    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033632    4860 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033635    4860 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033638    4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033641    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033644    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033647    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033650    4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033653    4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033655    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033658    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033661    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033663    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033666    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033669    4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033671    4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033674    4860 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033677    4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033679    4860 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:18.033693 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033682    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033684    4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033687    4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033694    4860 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033697    4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033700    4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033703    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033706    4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033708    4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033711    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033714    4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033716    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033719    4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033722    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033724    4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033727    4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033730    4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033732    4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033735    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033738    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:18.034216 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033741    4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033743    4860 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033746    4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033748    4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033751    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033754    4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033757    4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033759    4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033763    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033765    4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033768    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033771    4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033773    4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033776    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033779    4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033785    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033788    4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033791    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033794    4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033796    4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:18.034727 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033799    4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033801    4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033804    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033806    4860 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033809    4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033811    4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033816    4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033818    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033821    4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033824    4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033827    4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033830    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033832    4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033835    4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033838    4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033840    4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033843    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033845    4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033848    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:18.035241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033851    4860 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033853    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033856    4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033858    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033861    4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033863    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033866    4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033868    4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033872    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033875    4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033878    4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033880    4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033883    4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033885    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033888    4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033890    4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033893    4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033897    4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033901    4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:18.035726 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033906    4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033909    4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033912    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033915    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033918    4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033920    4860 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.033923    4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034671    4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034678    4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034681    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034684    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034686    4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034689    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034694    4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034697    4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034701    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034704    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034706    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034709    4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:18.036211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034711    4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034714    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034718    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034721    4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034723    4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034726    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034729    4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034732    4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034735    4860 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034737    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034740    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034742    4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034745    4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034748    4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034751    4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034754    4860 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034756    4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034760    4860 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034768    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034771    4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:18.036916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034774    4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034776    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034779    4860 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034781    4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034784    4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034786    4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034789    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034792    4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034794    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034796    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034799    4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034801    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034804    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034807    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034810    4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034814    4860 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034816    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034819    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034822    4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034824    4860 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:18.037438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034827    4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034830    4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034832    4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034835    4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034837    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034840    4860 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034842    4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034845    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034847    4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034850    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034852    4860 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034860    4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034863    4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034866    4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034868    4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034871    4860 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034874    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034877    4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034880    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034882    4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:18.037942 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034885    4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034887    4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034889    4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034892    4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034895    4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034897    4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034901    4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034904    4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034907    4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034910    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034912    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034915    4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034917    4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034920    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034922    4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034925    4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034927    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034930    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034932    4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:18.038447 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034937    4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034948    4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034952    4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034955    4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034958    4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034967    4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034970    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034973    4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034976    4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034978    4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034981    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034983    4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034986    4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034989    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.034992    4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035499    4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035504    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035507    4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035510    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:18.038968 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035514    4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035517    4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035520    4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035522    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035525    4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035527    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035530    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035533    4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035535    4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035539    4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035542    4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035544    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035547    4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035550    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035552    4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035555    4860 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035558    4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035560    4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035569    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:18.039470 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035573    4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035577    4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035579    4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035582    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035585    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035587    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035590    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035593    4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035595    4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035598    4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035600    4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035603    4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035605    4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035609    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035612    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035614    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035617    4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035620    4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:18.039950 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035622    4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035625    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035627    4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035630    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035634    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035637    4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035639    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035642    4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035645    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035647    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035650    4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035652    4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035655    4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035657    4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035667    4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035670    4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035674    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035676    4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035679    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035681    4860 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:18.040418 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035683    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035686    4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035688    4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035691    4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035693    4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035696    4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035698    4860 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035702    4860 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035705    4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035707    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035710    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035713    4860 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035715    4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035718    4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035721    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035723    4860 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035728    4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035731    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035733    4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035736    4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:18.040910 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035739    4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035741    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035744    4860 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035746    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035749    4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035752    4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035754    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035763    4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035767    4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035771    4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035774    4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035777    4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035780    4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035782    4860 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035785    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035787    4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035790    4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035796    4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035799    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:18.041422 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035802    4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035806    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035808    4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035810    4860 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035813    4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.035815    4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035923    4860 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035941    4860 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035951    4860 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035957    4860 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035961    4860 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035965    4860 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035969    4860 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035973    4860 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035976    4860 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035979    4860 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035982    4860 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035986    4860 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035989    4860 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035992    4860 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.035995    4860 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036004    4860 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036007    4860 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:14:18.041903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036010    4860 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036016    4860 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036019    4860 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036022    4860 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036025    4860 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036028    4860 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036032    4860 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036035    4860 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036038    4860 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036042    4860 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036045    4860 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036048    4860 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036051    4860 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036054    4860 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036057    4860 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036061    4860 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036064    4860 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036067    4860 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036073    4860 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036076    4860 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036079    4860 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036086    4860 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036090    4860 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036092    4860 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:14:18.042551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036096    4860 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036099    4860 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036103    4860 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036106    4860 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036109    4860 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036112    4860 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036115    4860 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036125    4860 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036129    4860 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036132    4860 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036135    4860 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036137    4860 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036140    4860 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036144    4860 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036148    4860 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036151    4860 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036155    4860 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036158    4860 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036161    4860 flags.go:64] FLAG: --help="false"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036164    4860 flags.go:64] FLAG: --hostname-override="ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036167    4860 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036170    4860 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036173    4860 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036176    4860 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:14:18.043489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036179    4860 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036182    4860 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036186    4860 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036189    4860 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036192    4860 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036195    4860 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036198    4860 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036201    4860 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036204    4860 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036207    4860 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036210    4860 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036213    4860 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036216    4860 flags.go:64] FLAG: --lock-file=""
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036219    4860 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036222    4860 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036225    4860 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036236    4860 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036239    4860 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036242    4860 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036245    4860 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036248    4860 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036251    4860 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036254    4860 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036257    4860 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036262    4860 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:14:18.044092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036265    4860 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036268    4860 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036271    4860 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036274    4860 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036277    4860 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036280    4860 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036283    4860 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036286    4860 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036289    4860 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036296    4860 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036299    4860 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036302    4860 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036305    4860 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036308    4860 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036311    4860 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036314    4860 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036318    4860 flags.go:64] FLAG: --port="10250"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036336    4860 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036341    4860 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a97299f08126dc40"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036346    4860 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036350    4860 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036353    4860 flags.go:64] FLAG: --register-node="true"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036356    4860 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036360    4860 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036369    4860 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036373    4860 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:14:18.044751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036375    4860 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036379    4860 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036382    4860 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036385    4860 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036388    4860 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036391    4860 flags.go:64] FLAG: --runonce="false"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036394    4860 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036397    4860 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036400    4860 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036403    4860 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036406    4860 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036409    4860 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036412    4860 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036415    4860 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036418    4860 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036423    4860 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036426    4860 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036429    4860 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036432    4860 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036435    4860 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036438    4860 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036443    4860 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036447    4860 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036450    4860 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036457    4860 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:14:18.045432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036460    4860 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036463    4860 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036465    4860 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036468    4860 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036471    4860 flags.go:64] FLAG: --v="2"
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036476    4860 flags.go:64] FLAG: --version="false"
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036485    4860 flags.go:64] FLAG: --vmodule=""
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036489    4860 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.036493    4860 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036682    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036691    4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036694    4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036697    4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036700    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036702    4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036705    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036709    4860 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036711    4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036714    4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036717    4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036720    4860 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036722    4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:18.046159 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036727    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036729    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036732    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036734    4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036737    4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036740    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036742    4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036746    4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036749    4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036752    4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036754    4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036757    4860 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036759    4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036762    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036765    4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036767    4860 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036770    4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036778    4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036781    4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036784    4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:18.046751 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036787    4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036789    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036792    4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036795    4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036799    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036803    4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036806    4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036808    4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036811    4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036813    4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036816    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036819    4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036823    4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036825    4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036828    4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036831    4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036835    4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036839    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:18.047302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036842    4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036846    4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036849    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036852    4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036854    4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036857    4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036859    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036862    4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036865    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036868    4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036870    4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036873    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036881    4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036883    4860 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036886    4860 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036889    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036891    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036894    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036897    4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036899    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:18.047866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036902    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036904    4860 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036907    4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036909    4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036912    4860 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036915    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036919    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036922    4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036925    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036928    4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036931    4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036933    4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036936    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036939    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036942    4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036945    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036947    4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036950    4860 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036952    4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036955    4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:18.048452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036958    4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036961    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036963    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036965    4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036968    4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036982    4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036985    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036987    4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036990    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036993    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036995    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.036998    4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037001    4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037003    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037006    4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.037032    4860 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:14:18.049035 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037081    4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037085    4860 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037090    4860 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037094    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037096    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037099    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037102    4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037104    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037107    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037109    4860 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037113    4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037116    4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037119    4860 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037121    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037125    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037127    4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037130    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037132    4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037135    4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037137    4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:18.049498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037140    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037143    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037151    4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037154    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037157    4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037159    4860 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037162    4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037165    4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037167    4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037170    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037172    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037175    4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037178    4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037180    4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037185    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037187    4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037190    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037192    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037195    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037197    4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:18.050135 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037200    4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037203    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037206    4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037209    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037212    4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037214    4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037217    4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037220    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037222    4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037225    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037228    4860 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037230    4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037233    4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037236    4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037238    4860 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037246    4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037255    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037258    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037261    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037263    4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:18.050851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037266    4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037268    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037271    4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037274    4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037276    4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037279    4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037282    4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037285    4860 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037287    4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037290    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037292    4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037295    4860 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037298    4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037300    4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037305    4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037308    4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037311    4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037313    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037316    4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:18.051414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037333    4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037337    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037340    4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037343    4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037346    4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037348    4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037351    4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037353    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037356    4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037364    4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037367    4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037370    4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037372    4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037376    4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037380    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037383    4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037385    4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037388    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037391    4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:18.051888 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037395    4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037397    4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037400    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037403    4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037406    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037409    4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037412    4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.037414    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.041652    4860 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.041666    4860 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.041682    4860 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.041687    4860 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041754    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041759    4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041762    4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041765    4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041768    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041771    4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041773    4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041776    4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:18.052357 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041779    4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041782    4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041790    4860 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041793    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041796    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041799    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041801    4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041804    4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041806    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041809    4860 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041812    4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041815    4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041817    4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041821    4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041823    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041826    4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041828    4860 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041831    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041834    4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041837    4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:18.052891 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041840    4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041843    4860 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041845    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041848    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041851    4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041857    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041860    4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041863    4860 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041865    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041868    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041871    4860 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041874    4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041876    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041879    4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041882    4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041894    4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041897    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041900    4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041903    4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:18.053393 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041906    4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041909    4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041911    4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041914    4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041917    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041920    4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041923    4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041925    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041928    4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041930    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041933    4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041935    4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041938    4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041941    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041948    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041951    4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041955    4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041959    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041963    4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:18.053852 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041965    4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041968    4860 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041971    4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041975    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041978    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041981    4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041984    4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041987    4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041990    4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041993    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041996    4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.041999    4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042001    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042004    4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042006    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042009    4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042012    4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042014    4860 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042017    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:18.054438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042019    4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042022    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042024    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042027    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042030    4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042032    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042035    4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042038    4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042040    4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042043    4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042046    4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042048    4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042051    4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042054    4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042056    4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042059    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042062    4860 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042064    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042067    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:18.054900 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042069    4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042073    4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.042101    4860 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042561    4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042588    4860 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042594    4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042600    4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042604    4860 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042609    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042614    4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042618    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042623    4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042633    4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042638    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042642    4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042647    4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:18.055376 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042651    4860 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042655    4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042659    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042664    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042668    4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042673    4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042677    4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042681    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042686    4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042695    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042702    4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042708    4860 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042713    4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042718    4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042722    4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042727    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042731    4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042736    4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042740    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:18.055778 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042744    4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042748    4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042757    4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042762    4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042767    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042770    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042775    4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042779    4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042783    4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042788    4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042792    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042799    4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042806    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042811    4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042820    4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042824    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042829    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042834    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042838    4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:18.056244 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042842    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042846    4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042850    4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042854    4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042858    4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042862    4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042866    4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042870    4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042879    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042884    4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042888    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042892    4860 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042929    4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042935    4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042947    4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042952    4860 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042956    4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042962    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042967    4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042971    4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:18.056755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042976    4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042985    4860 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042989    4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042993    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.042997    4860 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043001    4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043006    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043010    4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043014    4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043019    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043023    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043027    4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043031    4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043040    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043044    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043049    4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043053    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043057    4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043061    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043065    4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:18.057278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043069    4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043074    4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043119    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043124    4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043128    4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043132    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043142    4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043146    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043161    4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043165    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043170    4860 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043175    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043179    4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043184    4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043188    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:18.057863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.043231    4860 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043857    4860 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043872    4860 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043876    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043879    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043883    4860 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043885    4860 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043888    4860 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043892    4860 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043896    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043900    4860 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043906    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043909    4860 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043912    4860 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043915    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043918    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043921    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043923    4860 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043926    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:14:18.058237 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043929    4860 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043931    4860 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043934    4860 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043937    4860 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043939    4860 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043942    4860 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043945    4860 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043948    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043950    4860 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043954    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043957    4860 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043959    4860 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043962    4860 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043965    4860 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043967    4860 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043970    4860 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043973    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043976    4860 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043979    4860 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043982    4860 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:14:18.058734 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043984    4860 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043987    4860 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043989    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043992    4860 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043995    4860 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.043997    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044000    4860 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044002    4860 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044005    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044008    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044011    4860 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044014    4860 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044017    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044019    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044022    4860 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044025    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044028    4860 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044032    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044035    4860 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:14:18.059228 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044039    4860 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044043    4860 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044047    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044049    4860 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044053    4860 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044058    4860 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044063    4860 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044068    4860 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044071    4860 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044074    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044076    4860 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044079    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044082    4860 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044084    4860 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044087    4860 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044090    4860 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044092    4860 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044095    4860 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044098    4860 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:14:18.059743 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044100    4860 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044104    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044107    4860 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044110    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044113    4860 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044115    4860 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044118    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044123    4860 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044126    4860 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044129    4860 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044131    4860 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044134    4860 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044137    4860 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044139    4860 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044142    4860 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044145    4860 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044147    4860 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044150    4860 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044153    4860 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:14:18.060189 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044157    4860 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044160    4860 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044163    4860 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044166    4860 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044169    4860 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044173    4860 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044176    4860 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044178    4860 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044181    4860 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044183    4860 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:18.044186    4860 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.044250    4860 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.045201    4860 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.045250    4860 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.045457    4860 server.go:1014] "Starting client certificate rotation"
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.045610    4860 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.045684    4860 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:08:37 +0000 UTC" deadline="2028-04-15 20:01:32.92743362 +0000 UTC"
Oct 29 22:14:18.060721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.045732    4860 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12813h47m14.881705096s"
Oct 29 22:14:18.061455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.045911    4860 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:14:18.061455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.046947    4860 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:14:18.061455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.048286    4860 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:14:18.061455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.049958    4860 log.go:25] "Validated CRI v1 image API"
Oct 29 22:14:18.061455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.050730    4860 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:14:18.061455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.052463    4860 fs.go:135] Filesystem UUIDs: map[46d5fa04-d99a-4652-b74c-cd747d20a41b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e8d78256-42ef-4420-8e89-2d24bb29ed06:/dev/nvme0n1p3]
Oct 29 22:14:18.061721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.052481    4860 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c8b2d1b8c54d8dacd521bba7922e4d71a65aad010fd7532554f3ead854fa929/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c8b2d1b8c54d8dacd521bba7922e4d71a65aad010fd7532554f3ead854fa929/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2491f2b3af47d9d9e0dd4179ca3d0ecffc898997a55367d2ff090a61eccfbbc7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2491f2b3af47d9d9e0dd4179ca3d0ecffc898997a55367d2ff090a61eccfbbc7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e8d034b86c896517d7333ff588c2fe0f1ec44b1d7660306d27d8802e3576605/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e8d034b86c896517d7333ff588c2fe0f1ec44b1d7660306d27d8802e3576605/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30b6f54569da4ddd21c89d61a36d340e2d809a1325ae145988564b22ea17fc88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30b6f54569da4ddd21c89d61a36d340e2d809a1325ae145988564b22ea17fc88/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38c03c40eb74d96c3e261aa2044718a84fc65b51bcf5b776c0419a5faead95ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38c03c40eb74d96c3e261aa2044718a84fc65b51bcf5b776c0419a5faead95ec/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a92538a2ac695ed07c930ed2920e9c8efde421eb38e43c4f231f4a2dd42963d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a92538a2ac695ed07c930ed2920e9c8efde421eb38e43c4f231f4a2dd42963d/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d13fbd069491dc9e4bf3dc0076510878209847c2817f6c67275947f9e146fe3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d13fbd069491dc9e4bf3dc0076510878209847c2817f6c67275947f9e146fe3/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a2437d7c6feb068ac8c90b800f394c221ef0f5c4815f2ad8481d0bb28b9d491/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a2437d7c6feb068ac8c90b800f394c221ef0f5c4815f2ad8481d0bb28b9d491/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ca54851dd320d460ba4b60f33b7050ac1630d97a10d399d90c6362a80b98572/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ca54851dd320d460ba4b60f33b7050ac1630d97a10d399d90c6362a80b98572/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d4667d86df6e0ddc6785d2c7da6c7eac131b7c3f3d5f132c353d75d13ea9fee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d4667d86df6e0ddc6785d2c7da6c7eac131b7c3f3d5f132c353d75d13ea9fee/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7340e678fa66b915b570522eda264d7ee075877a3034baafcbf53d9c53a82a81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7340e678fa66b915b570522eda264d7ee075877a3034baafcbf53d9c53a82a81/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7bdf80363cc8ebc6e87f026eb86af0a1a09217f9a7e1bd59ec015198b9f5b1fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7bdf80363cc8ebc6e87f026eb86af0a1a09217f9a7e1bd59ec015198b9f5b1fd/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/802fc36f8fd0685143ad78ca1fe6300970f64a07df86cac38be2f0ad3c1bff9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/802fc36f8fd0685143ad78ca1fe6300970f64a07df86cac38be2f0ad3c1bff9e/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90b12b994e0a8646661226fd42f6773cb7cd595b40ea72558fed477065377385/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90b12b994e0a8646661226fd42f6773cb7cd595b40ea72558fed477065377385/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9214ce184041408a36b09d343d6ed2817bef97abef2c4fbb12f635796b8481b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9214ce184041408a36b09d343d6ed2817bef97abef2c4fbb12f635796b8481b6/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2423e0976b11dd60f5a69fcc2d098887e6bf9045380e7e8d71b6922462fea73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2423e0976b11dd60f5a69fcc2d098887e6bf9045380e7e8d71b6922462fea73/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b807268e76cb9ef3e16b3b00e571b3334acfba840b9008af81982896d2ccabbe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b807268e76cb9ef3e16b3b00e571b3334acfba840b9008af81982896d2ccabbe/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9de54c14081f90006f6c7c23ddf71f458c547b93db4173d1bdd04ab66e3cea0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9de54c14081f90006f6c7c23ddf71f458c547b93db4173d1bdd04ab66e3cea0/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbbf4631a66f7278c8564c3a9b7f6872cc908ac3f8e8a78f82d6562e4b8d70a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbbf4631a66f7278c8564c3a9b7f6872cc908ac3f8e8a78f82d6562e4b8d70a0/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e228425838c767c3d51e8a7849ae3082ef6bc00e11ba60b2058d289ba906a452/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e228425838c767c3d51e8a7849ae3082ef6bc00e11ba60b2058d289ba906a452/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2c7eecb67826582f9af117bab16905f4d8e24ea31d610a970e65b890b2f90b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2c7eecb67826582f9af117bab16905f4d8e24ea31d610a970e65b890b2f90b7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f898c7635d1499a52e6eaa271ab9c34e5cbe954e7499ea2de69e97d38096b4ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f898c7635d1499a52e6eaa271ab9c34e5cbe954e7499ea2de69e97d38096b4ad/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0605dc25-2004-42f2-a856-2f0be8e75ca2/volumes/kubernetes.io~projected/kube-api-access-hkhfw:{mountpoint:/var/lib/kubelet/pods/0605dc25-2004-42f2-a856-2f0be8e75ca2/volumes/kubernetes.io~projected/kube-api-access-hkhfw major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes/kubernetes.io~projected/kube-api-access-gwxqn:{mountpoint:/var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes/kubernetes.io~projected/kube-api-access-gwxqn major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19e92d72-5f01-4c0f-9739-e094b74d5825/volumes/kubernetes.io~projected/kube-api-access-972vc:{mountpoint:/var/lib/kubelet/pods/19e92d72-5f01-4c0f-9739-e094b74d5825/volumes/kubernetes.io~projected/kube-api-access-972vc major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19e92d72-5f01-4c0f-9739-e094b74d5825/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/19e92d72-5f01-4c0f-9739-e094b74d5825/volumes/kubernetes.io~secret/default-certificate major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19e92d72-5f01-4c0f-9739-e094b74d5825/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/19e92d72-5f01-4c0f-9739-e094b74d5825/volumes/kubernetes.io~secret/stats-auth major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e6952c3-e47b-4c03-8804-04de63d8f1ee/volumes/kubernetes.io~projected/kube-api-access-ljh6n:{mountpoint:/var/lib/kubelet/pods/1e6952c3-e47b-4c03-8804-04de63d8f1ee/volumes/kubernetes.io~projected/kube-api-access-ljh6n major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31bc5aff-808a-4d45-bcac-80d1bd626d47/volumes/kubernetes.io~projected/kube-api-access-l65cw:{mountpoint:/var/lib/kubelet/pods/31bc5aff-808a-4d45-bcac-80d1bd626d47/volumes/kubernetes.io~projected/kube-api-access-l65cw major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3611f6fc-6112-4b2e-8e22-75672c63c673/volumes/kubernetes.io~projected/kube-api-access-qk9z5:{mountpoint:/var/lib/kubelet/pods/3611f6fc-6112-4b2e-8e22-75672c63c673/volumes/kubernetes.io~projected/kube-api-access-qk9z5 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/529f43fd-d894-45ac-9143-2dc520293e16/volumes/kubernetes.io~projected/kube-api-access-vntmn:{mountpoint:/var/lib/kubelet/pods/529f43fd-d894-45ac-9143-2dc520293e16/volumes/kubernetes.io~projected/kube-api-access-vntmn major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/529f43fd-d894-45ac-9143-2dc520293e16/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/529f43fd-d894-45ac-9143-2dc520293e16/volumes/kubernetes.io~secret/cert major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5345c4bc-ec8a-454b-83bf-363e0d1432e6/volumes/kubernetes.io~projected/kube-api-access-rrmzq:{mountpoint:/var/lib/kubelet/pods/5345c4bc-ec8a-454b-83bf-363e0d1432e6/volumes/kubernetes.io~projected/kube-api-access-rrmzq major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c23cb73-3b67-46d7-94ab-ad2c0d9473de/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5c23cb73-3b67-46d7-94ab-ad2c0d9473de/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c23cb73-3b67-46d7-94ab-ad2c0d9473de/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5c23cb73-3b67-46d7-94ab-ad2c0d9473de/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c23cb73-3b67-46d7-94ab-ad2c0d9473de/volumes/kubernetes.io~projected/kube-api-access-zjgg2:{mountpoint:/var/lib/kubelet/pods/5c23cb73-3b67-46d7-94ab-ad2c0d9473de/volumes/kubernetes.io~projected/kube-api-access-zjgg2 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61824bf7-eca8-4554-8c54-81153bdcca82/volumes/kubernetes.io~projected/kube-api-access-tgwps:{mountpoint:/var/lib/kubelet/pods/61824bf7-eca8-4554-8c54-81153bdcca82/volumes/kubernetes.io~projected/kube-api-access-tgwps major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f6f3621-6d10-4afb-9438-c73cc3d4b13d/volumes/kubernetes.io~projected/kube-api-access-dplzf:{mountpoint:/var/lib/kubelet/pods/6f6f3621-6d10-4afb-9438-c73cc3d4b13d/volumes/kubernetes.io~projected/kube-api-access-dplzf major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7099dc88-0238-49a5-b0fb-8d008e1b040e/volumes/kubernetes.io~projected/kube-api-access-5d5rt:{mountpoint:/var/lib/kubelet/pods/7099dc88-0238-49a5-b0fb-8d008e1b040e/volumes/kubernetes.io~projected/kube-api-access-5d5rt major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7496f437-cb3c-4867-af37-16835f1134a1/volumes/kubernetes.io~projected/kube-api-access-bkw58:{mountpoint:/var/lib/kubelet/pods/7496f437-cb3c-4867-af37-16835f1134a1/volumes/kubernetes.io~projected/kube-api-access-bkw58 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7496f437-cb3c-4867-af37-16835f1134a1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/7496f437-cb3c-4867-af37-16835f1134a1/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d56328b-9fa0-45cb-996f-65c1c1b8785c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7d56328b-9fa0-45cb-996f-65c1c1b8785c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e0ef4de-c402-4c07-92f9-a7ec7d263113/volumes/kubernetes.io~projected/kube-api-access-2g29f:{mountpoint:/var/lib/kubelet/pods/8e0ef4de-c402-4c07-92f9-a7ec7d263113/volumes/kubernetes.io~projected/kube-api-access-2g29f major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93d782aa-ad96-4f81-a5a1-d356d022106b/volumes/kubernetes.io~projected/kube-api-access-d7drt:{mountpoint:/var/lib/kubelet/pods/93d782aa-ad96-4f81-a5a1-d356d022106b/volumes/kubernetes.io~projected/kube-api-access-d7drt major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f465b95-5af5-4d92-921b-24b0800934a0/volumes/kubernetes.io~projected/kube-api-access-57h7r:{mountpoint:/var/lib/kubelet/pods/9f465b95-5af5-4d92-921b-24b0800934a0/volumes/kubernetes.io~projected/kube-api-access-57h7r major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f465b95-5af5-4d92-921b-24b0800934a0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9f465b95-5af5-4d92-921b-24b0800934a0/volumes/kubernetes.io~secret/serving-cert major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a342967b-704c-4947-92fa-e2dc475675a0/volumes/kubernetes.io~projected/kube-api-access-6pkdx:{mountpoint:/var/lib/kubelet/pods/a342967b-704c-4947-92fa-e2dc475675a0/volumes/kubernetes.io~projected/kube-api-access-6pkdx major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a342967b-704c-4947-92fa-e2dc475675a0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a342967b-704c-4947-92fa-e2dc475675a0/volumes/kubernetes.io~secret/serving-cert major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4c49406-6e72-4bdd-8375-20dffea48c35/volumes/kubernetes.io~projected/kube-api-access-68rgp:{mountpoint:/var/lib/kubelet/pods/a4c49406-6e72-4bdd-8375-20dffea48c35/volumes/kubernetes.io~projected/kube-api-access-68rgp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b/volumes/kubernetes.io~projected/kube-api-access-6qdlv:{mountpoint:/var/lib/kubelet/pods/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b/volumes/kubernetes.io~projected/kube-api-access-6qdlv major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6c8fff1-c08b-4437-944f-9ee33d186c42/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b6c8fff1-c08b-4437-944f-9ee33d186c42/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b822897d-559d-47b4-b9e8-9ba7f417a7bd/volumes/kubernetes.io~projected/kube-api-access-8k5dd:{mountpoint:/var/lib/kubelet/pods/b822897d-559d-47b4-b9e8-9ba7f417a7bd/volumes/kubernetes.io~projected/kube-api-access-8k5dd major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccb78139-7ccc-4ac8-87bf-264d048488c4/volumes/kubernetes.io~projected/kube-api-access-52gp6:{mountpoint:/var/lib/kubelet/pods/ccb78139-7ccc-4ac8-87bf-264d048488c4/volumes/kubernetes.io~projected/kube-api-access-52gp6 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes/kubernetes.io~projected/kube-api-access-tg246:{mountpoint:/var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes/kubernetes.io~projected/kube-api-access-tg246 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df9ac5e7-84bd-4b61-958b-f78cd316e348/volumes/kubernetes.io~projected/kube-api-access-hbdg5:{mountpoint:/var/lib/kubelet/pods/df9ac5e7-84bd-4b61-958b-f78cd316e348/volumes/kubernetes.io~projected/kube-api-access-hbdg5 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e21cb68b-ed42-4009-9bf6-9700a98ffd42/volumes/kubernetes.io~projected/kube-api-access-mht74:{mountpoint:/var/lib/kubelet/pods/e21cb68b-ed42-4009-9bf6-9700a98ffd42/volumes/kubernetes.io~projected/kube-api-access-mht74 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb9f947e-2e34-4b91-9e2b-1edafb918fef/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/eb9f947e-2e34-4b91-9e2b-1edafb918fef/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb9f947e-2e34-4b91-9e2b-1edafb918fef/volumes/kubernetes.io~projected/kube-api-access-76cvd:{mountpoint:/var/lib/kubelet/pods/eb9f947e-2e34-4b91-9e2b-1edafb918fef/volumes/kubernetes.io~projected/kube-api-access-76cvd major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb9f947e-2e34-4b91-9e2b-1edafb918fef/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/eb9f947e-2e34-4b91-9e2b-1edafb918fef/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d/volumes/kubernetes.io~projected/kube-api-access-r86gd:{mountpoint:/var/lib/kubelet/pods/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d/volumes/kubernetes.io~projected/kube-api-access-r86gd major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb84513d-ca51-4920-bfc5-4240aa395064/volumes/kubernetes.io~projected/kube-api-access-w5pzj:{mountpoint:/var/lib/kubelet/pods/fb84513d-ca51-4920-bfc5-4240aa395064/volumes/kubernetes.io~projected/kube-api-access-w5pzj major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb84513d-ca51-4920-bfc5-4240aa395064/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fb84513d-ca51-4920-bfc5-4240aa395064/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c8d126ce9893ebf62416d05f194a8170c187291194dc4cc622c469ee99053b68/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9b8e1d0810ec25254af1d2ae6c37664cab21ac6d182c690e2f0e3a5b77e6084b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/cd18f8c33b6aed195db0f4711180588c9e049987cc214973c23df84a2341e8e1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/42939f7e7d63e5399549275eb02e3feb74777987e571d13ce80bc44a261e0c0f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/236d19df34850d64b361f0bdb94e33af98074fdca59c1bec18dd3eb5bcadfded/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1fc617a8c7f1915bee927da1e5eed46f88a2e30c26cd9c0a1ec0acf88e6c5520/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6f4c60e5af74e2ec26ce442b51fb1400b00390b759a2c3bed8cd5ebaf4b8acb0/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/0ff7213cf045f027e97627dadd4225757471589b9e40a74c2c3e1282fa7343e5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/f6bc5df9cc5a0b6fbdb543119a3f7609f78f4808fa28d17576e0e7b15af118b7/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/bf76f6fa1674bdeefb5863a598a67aba541c13160e9f7ffba2573b7d22e5a20e/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d33687bfe411e559b5ed0469b4f5ead68a500f8451aa75a21e6ef36ce334d7cb/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/358c969508f3a6f3d5920c3606312ac7d5daae8460cc8d8c85b1a7c16c00e26d/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c642c7610d7a601c06340574b79ba0690994e12c026ea1061793ff2129750c5e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f6da3135e0f4351230c90ce78d53b0643fd724d5e4ef71fa424a5abd9bb875c0/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a748186d6b848cb8d3989b465002ca7bf77c9f8ef43fe8cb689392993b247b3c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2b209afd6b846918c560c8806762028fffc11a64c4a9cde3f7423620b8d9d49c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/fca4652fd161ce374276035764f23736f2500c804e260f779e8a6b5c7157645e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/98b6410344581bdb4c80de769b37ce99e60f68530364de0eb76ff95f044eef83/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3234065c437634c447ea4e839cd9808ac31f175f4f304e4961915ba6cddc12e8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c9842a076dc7d4ddb6b914d4f799d1632d78af651ee8535ec57dfab6f789ce47/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/636f31e3472a062d1a1a054fee4232586bbf0aeac85a9666c52f70abb45e01d4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5c62b0bc0d1deacc1d543c86afcc1c3193d2db5d504a0b271638af256aa097cd/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/fc8b8dcc74f08dc50dd6709fdc39df33b5a3334b5f3f5a41da188ba3c2127ec4/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/ce7b9899eed93b037a79fd7292b51f2fc57f40106d829aaf17a0803c82faed14/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/d7b28290431d30799bf482ca54658541d983f1acf8e3d03b69c59917708e4419/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/08c4063b45cc4870782aff78f54d4eda1f06c2c81679510a3d1bd3b171a984fe/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/3f7e62b91208ab2ecf4b0e26ac4c75b23eddaf30f6d4b31bf87f11b26f319544/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/a3cb9bae8b0bcfb5eb3db2a6b124f4aee56dace559d9446394ac490f95ed12d9/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/9f78a3ce3e56de99c543a99c7f7e8dbb033343b9a59ffacfce022cd9ec25638b/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/c9015339916df1d1c602601de69030e1f5034096d3aa9e23234fe3a4fdaeca2e/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/bca8d003e91647e3a690c8c4d8e7337f2ad85bab9994bd0af115655a3ba56f92/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/024a496f9ef43e8359a5e2d4e532f07353d0da88d8a3228bc50d959960a68ba2/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/88780771973b07422e454b054d720aa867aaadf746ece2bf22a6a7009c9f8470/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/a0bc6df9f2d4193366d151a9fd4459a9f087e914883e80838894d1021b901fb6/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/20d0998a938ebf9290894cbeb5fe7c692d18098947986ffbd7ce0d89aeafad38/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/dc3ab31c2b966ac8e0917a58a9ce23d90de8f7a01e6c9f7da46d316c62d0ad62/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/4d488da88ae708724f772a3bf83d6b61f7a64bd72064f0c2b6d676d73904635f/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/f3fca2c9250e3d3e733b2f1bad1a218d25aeda1bcb22776fd614e7c1bf1fa940/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/0eaf86012905011ba2c27330b6fe31b783ed3e48b4e3ebdf86eaee39fb673552/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/3946c2b576beb09befe299cb0b78efb691a6bfc50ac2b83bffac08d9a3a6d2c1/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/de2523726642be28d5f795a7083e3d5798d17dea9bb941694cf723e68fe54c7e/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/e0d3df6c4d334a48ff4bd298f48dd3e4bb05bbdba6d021dce45b33bb2e42424f/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/0e42e8030516ea753c0898e52a79eb8a9aca67e2e4c4cb11de7f267f721d714b/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/16d24715a3898981a401d6821af087e479c21a52455da6cea8dc2de49f587665/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/54987add73afa1d30efaace79bfeb7bec7c94716e5c4ad30488b7479b86ef212/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b3793583401823fcc7f519c94fa74772727807a7d2bd4774361f1c07fffcb31c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a552a72344624e1279b85f5deee0044a26f614b81d71c419c3ebc66f7b0e9c9c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c9ad015b77f814bf26f478d46d8718070ccf10abc1e43258d5d48383d647eaec/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/886baa5681b0c44d83cb9fb4ca6816480b3c3f373fd26f7eedd2c48066334528/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/eda2966c5e9d219796954cdaf4efbe1977393fe116f1e51b41272bf54e62c537/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/11156cc240b23b68ca5126ef5aef0fae55d711c4d6c6b13ec703af2ceaec5d9f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6c2bf4434c81f624fe3cfd559fee820960fcc279db20ec7141e017fae9f0ad60/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1779a72bc93e7e801956386543434183f64470131ac7e8b7b81c704d594591ed/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d66f1b76c2ecf55388b6fc276d98b766a1fb7921163499b5079580d86dbb418b/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 29 22:14:18.061996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.060220    4860 manager.go:217] Machine: {Timestamp:2026-10-29 22:14:18.059776311 +0000 UTC m=+0.084552433 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098488 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec277aa0c2c13a13ec9371816e96cc62 SystemUUID:ec277aa0-c2c1-3a13-ec93-71816e96cc62 BootID:ae1e5626-b3c2-45d1-9718-833302044535 Filesystems:[{Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb9f947e-2e34-4b91-9e2b-1edafb918fef/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31bc5aff-808a-4d45-bcac-80d1bd626d47/volumes/kubernetes.io~projected/kube-api-access-l65cw DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df9ac5e7-84bd-4b61-958b-f78cd316e348/volumes/kubernetes.io~projected/kube-api-access-hbdg5 DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ca54851dd320d460ba4b60f33b7050ac1630d97a10d399d90c6362a80b98572/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4c49406-6e72-4bdd-8375-20dffea48c35/volumes/kubernetes.io~projected/kube-api-access-68rgp DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38c03c40eb74d96c3e261aa2044718a84fc65b51bcf5b776c0419a5faead95ec/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb84513d-ca51-4920-bfc5-4240aa395064/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7496f437-cb3c-4867-af37-16835f1134a1/volumes/kubernetes.io~projected/kube-api-access-bkw58 DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90b12b994e0a8646661226fd42f6773cb7cd595b40ea72558fed477065377385/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7340e678fa66b915b570522eda264d7ee075877a3034baafcbf53d9c53a82a81/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9f465b95-5af5-4d92-921b-24b0800934a0/volumes/kubernetes.io~projected/kube-api-access-57h7r DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f898c7635d1499a52e6eaa271ab9c34e5cbe954e7499ea2de69e97d38096b4ad/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5c23cb73-3b67-46d7-94ab-ad2c0d9473de/volumes/kubernetes.io~projected/kube-api-access-zjgg2 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3611f6fc-6112-4b2e-8e22-75672c63c673/volumes/kubernetes.io~projected/kube-api-access-qk9z5 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e228425838c767c3d51e8a7849ae3082ef6bc00e11ba60b2058d289ba906a452/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes/kubernetes.io~projected/kube-api-access-gwxqn DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5345c4bc-ec8a-454b-83bf-363e0d1432e6/volumes/kubernetes.io~projected/kube-api-access-rrmzq DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f6f3621-6d10-4afb-9438-c73cc3d4b13d/volumes/kubernetes.io~projected/kube-api-access-dplzf DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a342967b-704c-4947-92fa-e2dc475675a0/volumes/kubernetes.io~projected/kube-api-access-6pkdx DeviceMajor:0 DeviceMinor:235 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9de54c14081f90006f6c7c23ddf71f458c547b93db4173d1bdd04ab66e3cea0/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/802fc36f8fd0685143ad78ca1fe6300970f64a07df86cac38be2f0ad3c1bff9e/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19e92d72-5f01-4c0f-9739-e094b74d5825/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93d782aa-ad96-4f81-a5a1-d356d022106b/volumes/kubernetes.io~projected/kube-api-access-d7drt DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2c7eecb67826582f9af117bab16905f4d8e24ea31d610a970e65b890b2f90b7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d4667d86df6e0ddc6785d2c7da6c7eac131b7c3f3d5f132c353d75d13ea9fee/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb9f947e-2e34-4b91-9e2b-1edafb918fef/volumes/kubernetes.io~projected/kube-api-access-76cvd DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e0ef4de-c402-4c07-92f9-a7ec7d263113/volumes/kubernetes.io~projected/kube-api-access-2g29f DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbbf4631a66f7278c8564c3a9b7f6872cc908ac3f8e8a78f82d6562e4b8d70a0/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19e92d72-5f01-4c0f-9739-e094b74d5825/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9214ce184041408a36b09d343d6ed2817bef97abef2c4fbb12f635796b8481b6/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/61824bf7-eca8-4554-8c54-81153bdcca82/volumes/kubernetes.io~projected/kube-api-access-tgwps DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d56328b-9fa0-45cb-996f-65c1c1b8785c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0605dc25-2004-42f2-a856-2f0be8e75ca2/volumes/kubernetes.io~projected/kube-api-access-hkhfw DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a92538a2ac695ed07c930ed2920e9c8efde421eb38e43c4f231f4a2dd42963d/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7496f437-cb3c-4867-af37-16835f1134a1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b822897d-559d-47b4-b9e8-9ba7f417a7bd/volumes/kubernetes.io~projected/kube-api-access-8k5dd DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e21cb68b-ed42-4009-9bf6-9700a98ffd42/volumes/kubernetes.io~projected/kube-api-access-mht74 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b807268e76cb9ef3e16b3b00e571b3334acfba840b9008af81982896d2ccabbe/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/529f43fd-d894-45ac-9143-2dc520293e16/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:238 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/529f43fd-d894-45ac-9143-2dc520293e16/volumes/kubernetes.io~projected/kube-api-access-vntmn DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ccb78139-7ccc-4ac8-87bf-264d048488c4/volumes/kubernetes.io~projected/kube-api-access-52gp6 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19e92d72-5f01-4c0f-9739-e094b74d5825/volumes/kubernetes.io~projected/kube-api-access-972vc DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7099dc88-0238-49a5-b0fb-8d008e1b040e/volumes/kubernetes.io~projected/kube-api-access-5d5rt DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c23cb73-3b67-46d7-94ab-ad2c0d9473de/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c8b2d1b8c54d8dacd521bba7922e4d71a65aad010fd7532554f3ead854fa929/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d/volumes/kubernetes.io~projected/kube-api-access-r86gd DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb84513d-ca51-4920-bfc5-4240aa395064/volumes/kubernetes.io~projected/kube-api-access-w5pzj DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30b6f54569da4ddd21c89d61a36d340e2d809a1325ae145988564b22ea17fc88/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f465b95-5af5-4d92-921b-24b0800934a0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes/kubernetes.io~projected/kube-api-access-tg246 DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6c8fff1-c08b-4437-944f-9ee33d186c42/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb9f947e-2e34-4b91-9e2b-1edafb918fef/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e6952c3-e47b-4c03-8804-04de63d8f1ee/volumes/kubernetes.io~projected/kube-api-access-ljh6n DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a2437d7c6feb068ac8c90b800f394c221ef0f5c4815f2ad8481d0bb28b9d491/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b/volumes/kubernetes.io~projected/kube-api-access-6qdlv DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2491f2b3af47d9d9e0dd4179ca3d0ecffc898997a55367d2ff090a61eccfbbc7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5c23cb73-3b67-46d7-94ab-ad2c0d9473de/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a342967b-704c-4947-92fa-e2dc475675a0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e8d034b86c896517d7333ff588c2fe0f1ec44b1d7660306d27d8802e3576605/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d13fbd069491dc9e4bf3dc0076510878209847c2817f6c67275947f9e146fe3/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2423e0976b11dd60f5a69fcc2d098887e6bf9045380e7e8d71b6922462fea73/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7bdf80363cc8ebc6e87f026eb86af0a1a09217f9a7e1bd59ec015198b9f5b1fd/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1c8b2d1b8c54d8d MacAddress:d6:53:fc:83:81:df Speed:10000 Mtu:8901} {Name:2e8d034b86c8965 MacAddress:1e:bd:93:82:b8:07 Speed:10000 Mtu:8901} {Name:30b6f54569da4dd MacAddress:9a:9c:8e:60:a4:dc Speed:10000 Mtu:8901} {Name:4a2437d7c6feb06 MacAddress:fa:7f:f9:61:a1:8f Speed:10000 Mtu:8901} {Name:6d4667d86df6e0d MacAddress:52:ea:bb:46:b4:07 Speed:10000 Mtu:8901} {Name:802fc36f8fd0685 MacAddress:e6:0f:1a:17:75:8f Speed:10000 Mtu:8901} {Name:90b12b994e0a864 MacAddress:42:14:fc:3e:d7:48 Speed:10000 Mtu:8901} {Name:9214ce184041408 MacAddress:be:9d:f0:89:a9:69 Speed:10000 Mtu:8901} {Name:a2423e0976b11dd MacAddress:8e:bc:f0:cb:b0:77 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f9:b3:1c:7d Speed:0 Mtu:9001} {Name:br-int MacAddress:b2:38:2f:06:ac:4d Speed:0 Mtu:8901} {Name:c9de54c14081f90 MacAddress:5a:59:67:61:cb:b8 Speed:10000 Mtu:8901} {Name:cbbf4631a66f727 MacAddress:82:7b:c3:15:31:43 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f9:b3:1c:7d Speed:0 Mtu:9001} {Name:f898c7635d1499a MacAddress:0a:c8:91:08:f4:45 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:3e:c8:fd:71:ff:d0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:82:a4:0c:6f:0f:7d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:14:18.061996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.060653    4860 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:14:18.061996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.060727    4860 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:14:18.061996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.061041    4860 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:14:18.061996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.061062    4860 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:14:18.061996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.061372    4860 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:14:18.061996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.061385    4860 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:14:18.061996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.061399    4860 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:14:18.061996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.061416    4860 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:14:18.061996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.061442    4860 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:14:18.061996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.061753    4860 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:14:18.061996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.062001    4860 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:14:18.062617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.062041    4860 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:14:18.062617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.062060    4860 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:14:18.062617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.062075    4860 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:14:18.062617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.062088    4860 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:14:18.062617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.062098    4860 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:14:18.062985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.062970    4860 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:14:18.063022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.062987    4860 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:14:18.063516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.063496    4860 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:14:18.065521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065508    4860 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:14:18.065521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065520    4860 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:14:18.065615 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065606    4860 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 29 22:14:18.065783 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065773    4860 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:14:18.065813 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065801    4860 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:14:18.065889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065880    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:14:18.065922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065894    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:14:18.065922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065900    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:14:18.065922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065905    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:14:18.065922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065912    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:14:18.065922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065918    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:14:18.065922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065923    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:14:18.066072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065929    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:14:18.066072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065936    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:14:18.066072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065942    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:14:18.066072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065950    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:14:18.066072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065959    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:14:18.066072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065993    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:14:18.066072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.065999    4860 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:14:18.069234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.069222    4860 server.go:1263] "Started kubelet"
Oct 29 22:14:18.069393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.069341    4860 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:14:18.069743 ip-10-0-128-31 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:14:18.069868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.069348    4860 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:14:18.069868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.069832    4860 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:14:18.070136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.070117    4860 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:14:18.071046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.071028    4860 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:14:18.073936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.073917    4860 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:14:18.074301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.074284    4860 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:14:18.074301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.074300    4860 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:14:18.074537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.074424    4860 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:14:18.074537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.074296    4860 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:14:18.074537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.074473    4860 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:08:39 +0000 UTC" deadline="2028-06-23 21:12:27.224615004 +0000 UTC"
Oct 29 22:14:18.074537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.074500    4860 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14470h58m9.15011714s"
Oct 29 22:14:18.075070 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:18.075048    4860 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-31.ec2.internal\" not found"
Oct 29 22:14:18.076983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.076693    4860 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:14:18.076983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.076712    4860 factory.go:55] Registering systemd factory
Oct 29 22:14:18.076983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.076724    4860 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:14:18.077745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.077703    4860 factory.go:153] Registering CRI-O factory
Oct 29 22:14:18.077745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.077720    4860 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:14:18.077850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.077764    4860 factory.go:103] Registering Raw factory
Oct 29 22:14:18.077850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.077781    4860 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078117    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="529f43fd-d894-45ac-9143-2dc520293e16" volumeName="kubernetes.io/secret/529f43fd-d894-45ac-9143-2dc520293e16-cert" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078170    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df9ac5e7-84bd-4b61-958b-f78cd316e348" volumeName="kubernetes.io/empty-dir/df9ac5e7-84bd-4b61-958b-f78cd316e348-data-volume" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078186    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb9f947e-2e34-4b91-9e2b-1edafb918fef" volumeName="kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovnkube-script-lib" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078204    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb84513d-ca51-4920-bfc5-4240aa395064" volumeName="kubernetes.io/secret/fb84513d-ca51-4920-bfc5-4240aa395064-serving-cert" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078218    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7099dc88-0238-49a5-b0fb-8d008e1b040e" volumeName="kubernetes.io/projected/7099dc88-0238-49a5-b0fb-8d008e1b040e-kube-api-access-5d5rt" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078295    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0f94ef8-7c40-437b-a27e-36583be8bfba" volumeName="kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-trusted-ca" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078342    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e21cb68b-ed42-4009-9bf6-9700a98ffd42" volumeName="kubernetes.io/configmap/e21cb68b-ed42-4009-9bf6-9700a98ffd42-iptables-alerter-script" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078374    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb9f947e-2e34-4b91-9e2b-1edafb918fef" volumeName="kubernetes.io/secret/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078397    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f6f3621-6d10-4afb-9438-c73cc3d4b13d" volumeName="kubernetes.io/projected/6f6f3621-6d10-4afb-9438-c73cc3d4b13d-kube-api-access-dplzf" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078419    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e6952c3-e47b-4c03-8804-04de63d8f1ee" volumeName="kubernetes.io/projected/1e6952c3-e47b-4c03-8804-04de63d8f1ee-kube-api-access-ljh6n" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078428    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e92d72-5f01-4c0f-9739-e094b74d5825" volumeName="kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-default-certificate" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078437    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c23cb73-3b67-46d7-94ab-ad2c0d9473de" volumeName="kubernetes.io/empty-dir/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-tuned" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078449    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f465b95-5af5-4d92-921b-24b0800934a0" volumeName="kubernetes.io/configmap/9f465b95-5af5-4d92-921b-24b0800934a0-config" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078457    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b822897d-559d-47b4-b9e8-9ba7f417a7bd" volumeName="kubernetes.io/configmap/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-config" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078471    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb9f947e-2e34-4b91-9e2b-1edafb918fef" volumeName="kubernetes.io/projected/eb9f947e-2e34-4b91-9e2b-1edafb918fef-kube-api-access-76cvd" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078479    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31bc5aff-808a-4d45-bcac-80d1bd626d47" volumeName="kubernetes.io/projected/31bc5aff-808a-4d45-bcac-80d1bd626d47-kube-api-access-l65cw" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078489    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3611f6fc-6112-4b2e-8e22-75672c63c673" volumeName="kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078498    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a97d96fb-da2b-4064-b6bb-ac7f3cf2270b" volumeName="kubernetes.io/projected/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-kube-api-access-6qdlv" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078522    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df9ac5e7-84bd-4b61-958b-f78cd316e348" volumeName="kubernetes.io/projected/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-api-access-hbdg5" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078530    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e21cb68b-ed42-4009-9bf6-9700a98ffd42" volumeName="kubernetes.io/projected/e21cb68b-ed42-4009-9bf6-9700a98ffd42-kube-api-access-mht74" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078539    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d" volumeName="kubernetes.io/configmap/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-telemetry-config" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078551    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb84513d-ca51-4920-bfc5-4240aa395064" volumeName="kubernetes.io/empty-dir/fb84513d-ca51-4920-bfc5-4240aa395064-tmp" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078559    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da83403-18e0-4abd-8bbb-9323e80a3959" volumeName="kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-trusted-ca" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078570    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e92d72-5f01-4c0f-9739-e094b74d5825" volumeName="kubernetes.io/projected/19e92d72-5f01-4c0f-9739-e094b74d5825-kube-api-access-972vc" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078578    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4c49406-6e72-4bdd-8375-20dffea48c35" volumeName="kubernetes.io/projected/a4c49406-6e72-4bdd-8375-20dffea48c35-kube-api-access-68rgp" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078586    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c23cb73-3b67-46d7-94ab-ad2c0d9473de" volumeName="kubernetes.io/projected/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-kube-api-access-zjgg2" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078596    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a342967b-704c-4947-92fa-e2dc475675a0" volumeName="kubernetes.io/configmap/a342967b-704c-4947-92fa-e2dc475675a0-config" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078604    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccb78139-7ccc-4ac8-87bf-264d048488c4" volumeName="kubernetes.io/projected/ccb78139-7ccc-4ac8-87bf-264d048488c4-kube-api-access-52gp6" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078620    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb84513d-ca51-4920-bfc5-4240aa395064" volumeName="kubernetes.io/configmap/fb84513d-ca51-4920-bfc5-4240aa395064-trusted-ca-bundle" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078629    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da83403-18e0-4abd-8bbb-9323e80a3959" volumeName="kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-installation-pull-secrets" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078639    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7496f437-cb3c-4867-af37-16835f1134a1" volumeName="kubernetes.io/configmap/7496f437-cb3c-4867-af37-16835f1134a1-config" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078642    4860 manager.go:319] Starting recovery of all containers
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078648    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0f94ef8-7c40-437b-a27e-36583be8bfba" volumeName="kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-kube-api-access-tg246" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078659    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0f94ef8-7c40-437b-a27e-36583be8bfba" volumeName="kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-image-registry-private-configuration" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078667    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb84513d-ca51-4920-bfc5-4240aa395064" volumeName="kubernetes.io/projected/fb84513d-ca51-4920-bfc5-4240aa395064-kube-api-access-w5pzj" seLinuxMountContext=""
Oct 29 22:14:18.078185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.078676    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0605dc25-2004-42f2-a856-2f0be8e75ca2" volumeName="kubernetes.io/projected/0605dc25-2004-42f2-a856-2f0be8e75ca2-kube-api-access-hkhfw" seLinuxMountContext=""
Oct 29 22:14:18.080376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.079086    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3611f6fc-6112-4b2e-8e22-75672c63c673" volumeName="kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 29 22:14:18.081142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081123    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93d782aa-ad96-4f81-a5a1-d356d022106b" volumeName="kubernetes.io/configmap/93d782aa-ad96-4f81-a5a1-d356d022106b-serviceca" seLinuxMountContext=""
Oct 29 22:14:18.081208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081149    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f465b95-5af5-4d92-921b-24b0800934a0" volumeName="kubernetes.io/projected/9f465b95-5af5-4d92-921b-24b0800934a0-kube-api-access-57h7r" seLinuxMountContext=""
Oct 29 22:14:18.081208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081162    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a342967b-704c-4947-92fa-e2dc475675a0" volumeName="kubernetes.io/secret/a342967b-704c-4947-92fa-e2dc475675a0-serving-cert" seLinuxMountContext=""
Oct 29 22:14:18.081208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081174    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb84513d-ca51-4920-bfc5-4240aa395064" volumeName="kubernetes.io/configmap/fb84513d-ca51-4920-bfc5-4240aa395064-service-ca-bundle" seLinuxMountContext=""
Oct 29 22:14:18.081208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081186    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da83403-18e0-4abd-8bbb-9323e80a3959" volumeName="kubernetes.io/empty-dir/0da83403-18e0-4abd-8bbb-9323e80a3959-ca-trust-extracted" seLinuxMountContext=""
Oct 29 22:14:18.081208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081198    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3611f6fc-6112-4b2e-8e22-75672c63c673" volumeName="kubernetes.io/projected/3611f6fc-6112-4b2e-8e22-75672c63c673-kube-api-access-qk9z5" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081209    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7496f437-cb3c-4867-af37-16835f1134a1" volumeName="kubernetes.io/secret/7496f437-cb3c-4867-af37-16835f1134a1-serving-cert" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081220    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f465b95-5af5-4d92-921b-24b0800934a0" volumeName="kubernetes.io/secret/9f465b95-5af5-4d92-921b-24b0800934a0-serving-cert" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081232    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a342967b-704c-4947-92fa-e2dc475675a0" volumeName="kubernetes.io/projected/a342967b-704c-4947-92fa-e2dc475675a0-kube-api-access-6pkdx" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081245    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6c8fff1-c08b-4437-944f-9ee33d186c42" volumeName="kubernetes.io/configmap/b6c8fff1-c08b-4437-944f-9ee33d186c42-konnectivity-ca" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081256    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb9f947e-2e34-4b91-9e2b-1edafb918fef" volumeName="kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovnkube-config" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081268    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20f39049-d7a1-46d8-8ebc-c0159906d689" volumeName="kubernetes.io/configmap/20f39049-d7a1-46d8-8ebc-c0159906d689-nginx-conf" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081278    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7099dc88-0238-49a5-b0fb-8d008e1b040e" volumeName="kubernetes.io/empty-dir/7099dc88-0238-49a5-b0fb-8d008e1b040e-tmp-dir" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081291    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6c8fff1-c08b-4437-944f-9ee33d186c42" volumeName="kubernetes.io/secret/b6c8fff1-c08b-4437-944f-9ee33d186c42-agent-certs" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081303    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b822897d-559d-47b4-b9e8-9ba7f417a7bd" volumeName="kubernetes.io/projected/b822897d-559d-47b4-b9e8-9ba7f417a7bd-kube-api-access-8k5dd" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081313    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0f94ef8-7c40-437b-a27e-36583be8bfba" volumeName="kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-installation-pull-secrets" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081408    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61824bf7-eca8-4554-8c54-81153bdcca82" volumeName="kubernetes.io/projected/61824bf7-eca8-4554-8c54-81153bdcca82-kube-api-access-tgwps" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081432    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da83403-18e0-4abd-8bbb-9323e80a3959" volumeName="kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-kube-api-access-gwxqn" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081445    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da83403-18e0-4abd-8bbb-9323e80a3959" volumeName="kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-image-registry-private-configuration" seLinuxMountContext=""
Oct 29 22:14:18.081469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081457    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e0ef4de-c402-4c07-92f9-a7ec7d263113" volumeName="kubernetes.io/configmap/8e0ef4de-c402-4c07-92f9-a7ec7d263113-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081479    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d" volumeName="kubernetes.io/projected/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-kube-api-access-r86gd" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081491    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5345c4bc-ec8a-454b-83bf-363e0d1432e6" volumeName="kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081505    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0f94ef8-7c40-437b-a27e-36583be8bfba" volumeName="kubernetes.io/empty-dir/d0f94ef8-7c40-437b-a27e-36583be8bfba-ca-trust-extracted" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081517    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e0ef4de-c402-4c07-92f9-a7ec7d263113" volumeName="kubernetes.io/projected/8e0ef4de-c402-4c07-92f9-a7ec7d263113-kube-api-access-2g29f" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081530    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb84513d-ca51-4920-bfc5-4240aa395064" volumeName="kubernetes.io/empty-dir/fb84513d-ca51-4920-bfc5-4240aa395064-snapshots" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081542    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3611f6fc-6112-4b2e-8e22-75672c63c673" volumeName="kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081553    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c23cb73-3b67-46d7-94ab-ad2c0d9473de" volumeName="kubernetes.io/empty-dir/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-tmp" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081565    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7099dc88-0238-49a5-b0fb-8d008e1b040e" volumeName="kubernetes.io/configmap/7099dc88-0238-49a5-b0fb-8d008e1b040e-config-volume" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081577    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7496f437-cb3c-4867-af37-16835f1134a1" volumeName="kubernetes.io/configmap/7496f437-cb3c-4867-af37-16835f1134a1-trusted-ca" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081589    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7496f437-cb3c-4867-af37-16835f1134a1" volumeName="kubernetes.io/projected/7496f437-cb3c-4867-af37-16835f1134a1-kube-api-access-bkw58" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081600    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d56328b-9fa0-45cb-996f-65c1c1b8785c" volumeName="kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081613    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e92d72-5f01-4c0f-9739-e094b74d5825" volumeName="kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-stats-auth" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081623    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da83403-18e0-4abd-8bbb-9323e80a3959" volumeName="kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-bound-sa-token" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081636    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4c49406-6e72-4bdd-8375-20dffea48c35" volumeName="kubernetes.io/empty-dir/a4c49406-6e72-4bdd-8375-20dffea48c35-tmp-dir" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081648    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0f94ef8-7c40-437b-a27e-36583be8bfba" volumeName="kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-certificates" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081674    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da83403-18e0-4abd-8bbb-9323e80a3959" volumeName="kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-certificates" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081685    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="529f43fd-d894-45ac-9143-2dc520293e16" volumeName="kubernetes.io/projected/529f43fd-d894-45ac-9143-2dc520293e16-kube-api-access-vntmn" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081696    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e0ef4de-c402-4c07-92f9-a7ec7d263113" volumeName="kubernetes.io/configmap/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-daemon-config" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081708    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93d782aa-ad96-4f81-a5a1-d356d022106b" volumeName="kubernetes.io/projected/93d782aa-ad96-4f81-a5a1-d356d022106b-kube-api-access-d7drt" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081720    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0f94ef8-7c40-437b-a27e-36583be8bfba" volumeName="kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-bound-sa-token" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081731    4860 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb9f947e-2e34-4b91-9e2b-1edafb918fef" volumeName="kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-env-overrides" seLinuxMountContext=""
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081742    4860 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:14:18.082050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.081750    4860 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:14:18.093232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.093209    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/1.log"
Oct 29 22:14:18.093366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.093247    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/1.log"
Oct 29 22:14:18.101982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.101957    4860 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:14:18.103296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.103275    4860 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:14:18.103296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.103298    4860 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:14:18.103452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.103316    4860 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:14:18.103452 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:18.103376    4860 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 22:14:18.115072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.115047    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/1.log"
Oct 29 22:14:18.115160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.115086    4860 generic.go:356] "Generic (PLEG): container finished" podID="7496f437-cb3c-4867-af37-16835f1134a1" containerID="7c727fc6b491e3c7a5d226c6cccc83981ca0988ffe7eb5d016e5d71adef13b31" exitCode=255
Oct 29 22:14:18.125851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.125823    4860 generic.go:356] "Generic (PLEG): container finished" podID="1bb6269644393a710c6775a446a340ab" containerID="1acf379624438ddd0be82d11391dfc18fd2291e74ca790f56bcc8c92bc7d3247" exitCode=0
Oct 29 22:14:18.127077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.127053    4860 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:14:18.127199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.127178    4860 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:14:18.127309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.127292    4860 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 29 22:14:18.141411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.141379    4860 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:14:18.141706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.141681    4860 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:14:18.145556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.145533    4860 generic.go:356] "Generic (PLEG): container finished" podID="3611f6fc-6112-4b2e-8e22-75672c63c673" containerID="80b6e52b371ff5e2060bc0104fd1aec8be6b4b99516b7e3129263218bbac7ca3" exitCode=0
Oct 29 22:14:18.145556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.145554    4860 generic.go:356] "Generic (PLEG): container finished" podID="3611f6fc-6112-4b2e-8e22-75672c63c673" containerID="234b0b53b0e5ae9147a93313054f99ebc34dd84353019483d6bc55ca2c508539" exitCode=0
Oct 29 22:14:18.145684 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.145560    4860 generic.go:356] "Generic (PLEG): container finished" podID="3611f6fc-6112-4b2e-8e22-75672c63c673" containerID="8a1ded9624a6c0d1e06e1c199299cdf59b65b0496034ee229c9698868b8f165a" exitCode=0
Oct 29 22:14:18.145684 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.145565    4860 generic.go:356] "Generic (PLEG): container finished" podID="3611f6fc-6112-4b2e-8e22-75672c63c673" containerID="96492b43253ec6d0147a13224a15ad4f71e8aed1940ff973e2199a7843615ce6" exitCode=0
Oct 29 22:14:18.145684 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.145570    4860 generic.go:356] "Generic (PLEG): container finished" podID="3611f6fc-6112-4b2e-8e22-75672c63c673" containerID="c0059063d39f28606cf74561d0a69cc3d85ee5f009de816c01945234020cf868" exitCode=0
Oct 29 22:14:18.145684 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.145575    4860 generic.go:356] "Generic (PLEG): container finished" podID="3611f6fc-6112-4b2e-8e22-75672c63c673" containerID="45bd423c6d556e36d2f1ec58c952847192bd659bd06d51fadccdd3e23e4417a3" exitCode=0
Oct 29 22:14:18.168747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.168693    4860 manager.go:324] Recovery completed
Oct 29 22:14:18.186829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.186810    4860 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:14:18.186829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.186822    4860 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:14:18.186951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.186837    4860 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:14:18.186984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.186954    4860 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 29 22:14:18.186984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.186964    4860 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 29 22:14:18.186984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.186980    4860 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 29 22:14:18.187084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.186986    4860 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 29 22:14:18.187084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.186991    4860 policy_none.go:50] "Start"
Oct 29 22:14:18.187084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.186999    4860 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:14:18.187084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.187008    4860 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:14:18.187197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.187125    4860 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 29 22:14:18.187197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.187132    4860 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 29 22:14:18.187197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.187137    4860 policy_none.go:44] "Start"
Oct 29 22:14:18.191225 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.191200    4860 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:14:18.193678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.193666    4860 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:14:18.194119 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:18.193692    4860 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:14:18.194189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.194120    4860 server.go:87] "Starting device plugin registration server"
Oct 29 22:14:18.194407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.194394    4860 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:14:18.194482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.194409    4860 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:14:18.194533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.194500    4860 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:14:18.194635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.194598    4860 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:14:18.194732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.194659    4860 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:14:18.194732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.194667    4860 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:14:18.194843 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.194701    4860 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:14:18.194632679Z","UUID":"6f1ecc81-91a9-4087-bce3-1b097e99e9ef","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:14:18.196105 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:18.196070    4860 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:14:18.196562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.196545    4860 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:14:18.196652 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.196631    4860 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:14:18.203489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.203451    4860 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"]
Oct 29 22:14:18.204114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.204092    4860 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.204114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.204098    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" event={"ID":"ed322e80a89b2bbeafbf0f88fe9597d6","Type":"ContainerStarted","Data":"e9f2c0851e694203096b6731d4cacaa871aaa693db1fe52f6ffcdbd335d2e49e"}
Oct 29 22:14:18.204268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.204137    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal" event={"ID":"ed322e80a89b2bbeafbf0f88fe9597d6","Type":"ContainerStarted","Data":"5ca54851dd320d460ba4b60f33b7050ac1630d97a10d399d90c6362a80b98572"}
Oct 29 22:14:18.204268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.204156    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"1bb6269644393a710c6775a446a340ab","Type":"ContainerStarted","Data":"e04bfb33f6fe4adb6a7bcf38a1199ade15c8953b31cc293179bb69970bdd4b1a"}
Oct 29 22:14:18.204268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.204172    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"1bb6269644393a710c6775a446a340ab","Type":"ContainerDied","Data":"1acf379624438ddd0be82d11391dfc18fd2291e74ca790f56bcc8c92bc7d3247"}
Oct 29 22:14:18.204268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.204189    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal" event={"ID":"1bb6269644393a710c6775a446a340ab","Type":"ContainerStarted","Data":"2491f2b3af47d9d9e0dd4179ca3d0ecffc898997a55367d2ff090a61eccfbbc7"}
Oct 29 22:14:18.204502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.204319    4860 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.228191 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.228176    4860 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:18.228266 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:18.228217    4860 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.228893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.228881    4860 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:18.228932 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:18.228907    4860 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.295109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.295088    4860 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:14:18.296143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.296127    4860 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:14:18.296234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.296157    4860 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:14:18.296234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.296170    4860 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-31.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:14:18.296234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.296222    4860 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.327960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.327918    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.328041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.327984    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed322e80a89b2bbeafbf0f88fe9597d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"ed322e80a89b2bbeafbf0f88fe9597d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.328041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.328012    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.332201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.332186    4860 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.332269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.332231    4860 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.428718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.428664    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed322e80a89b2bbeafbf0f88fe9597d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"ed322e80a89b2bbeafbf0f88fe9597d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.428718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.428696    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.428849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.428723    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.428849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.428774    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.428849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.428777    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed322e80a89b2bbeafbf0f88fe9597d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" (UID: \"ed322e80a89b2bbeafbf0f88fe9597d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:18.428849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:18.428777    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb6269644393a710c6775a446a340ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" (UID: \"1bb6269644393a710c6775a446a340ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:19.062214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.062184    4860 apiserver.go:52] "Watching apiserver"
Oct 29 22:14:19.121566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.121544    4860 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:14:19.122234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.122212    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-75pjw","openshift-multus/network-metrics-daemon-l9rds","openshift-ingress-canary/ingress-canary-mld7c","openshift-ingress/router-default-7b6b64f55f-9l4xp","openshift-dns/node-resolver-m7gqm","kube-system/kas-connection-checker-685895c7c9-2z8f7","openshift-image-registry/image-registry-9f64d5bbb-8pnfd","kube-system/kas-connection-checker-685895c7c9-stxg2","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2","openshift-network-console/networking-console-plugin-86c478bb88-w9grl","openshift-network-diagnostics/network-check-target-sd9mh","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6","kube-system/global-pull-secret-syncer-stxld","openshift-multus/multus-additional-cni-plugins-xqtzn","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg","openshift-dns/dns-default-5sn5b","openshift-image-registry/image-registry-6f8746f6cf-j8g6f","openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6","kube-system/konnectivity-agent-99lmd","openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww","openshift-image-registry/node-ca-x8qdl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal","openshift-cluster-node-tuning-operator/tuned-mmk2r","openshift-insights/insights-operator-7c4986d449-wfpts","kube-system/kas-connection-checker-685895c7c9-6zmj5","openshift-insights/insights-runtime-extractor-pd9qn","openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5","openshift-network-operator/iptables-alerter-7lbzl","openshift-console-operator/console-operator-7cd56b8dbc-7gx9p","kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-774ht"]
Oct 29 22:14:19.126620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.126597    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.126955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.126924    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.127869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.127854    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:19.128549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.128530    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:19.129309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.129292    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:19.133259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.133244    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:19.135135 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.135117    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.136063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.136044    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:19.136978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.136961    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.140922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.140901    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.141000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.140941    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 29 22:14:19.142763 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.142745    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.142763 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.142758    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6sn8x\""
Oct 29 22:14:19.142887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.142755    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 29 22:14:19.142887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.142755    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.142887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.142755    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.142887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.142805    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.142887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.142809    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rzd6j\""
Oct 29 22:14:19.142887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.142872    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.144255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.144241    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 29 22:14:19.144660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.144644    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-s4dxs\""
Oct 29 22:14:19.145610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.145596    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 29 22:14:19.150950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.150928    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 29 22:14:19.152657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.152614    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.152657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.152628    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.152657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.152620    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.152657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.152625    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 29 22:14:19.152657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.152634    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.152930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.152672    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.152930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.152678    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 29 22:14:19.152930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.152678    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:14:19.152930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.152692    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 29 22:14:19.152930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.152697    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.152930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.152848    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n84hz\""
Oct 29 22:14:19.152930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.152863    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x9frc\""
Oct 29 22:14:19.156099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.156086    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 29 22:14:19.156383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.156368    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 29 22:14:19.156453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.156390    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 29 22:14:19.156453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.156404    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 29 22:14:19.156453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.156391    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 29 22:14:19.156556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.156465    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-k6pmr\""
Oct 29 22:14:19.157095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.157072    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hkdgm\""
Oct 29 22:14:19.157182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.157111    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cwsdq\""
Oct 29 22:14:19.158004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.157971    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r2ft9\""
Oct 29 22:14:19.158695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.158676    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 29 22:14:19.160058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.160033    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v7npq\""
Oct 29 22:14:19.160600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.160583    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7vbxc\""
Oct 29 22:14:19.160776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.160751    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w4b7h\""
Oct 29 22:14:19.160869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.160805    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 29 22:14:19.160869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.160849    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 29 22:14:19.161031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.161014    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:14:19.161112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.161092    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 29 22:14:19.161626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.161612    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2nqlz\""
Oct 29 22:14:19.161873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.161858    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bjh7c\""
Oct 29 22:14:19.162055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.162042    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xsn6t\""
Oct 29 22:14:19.162265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.162253    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 29 22:14:19.163413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.163400    4860 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:19.163523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.163511    4860 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:19.176057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.176039    4860 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:14:19.186031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.186018    4860 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:19.186102 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.186058    4860 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-31.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:19.189474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.189457    4860 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:14:19.189536 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.189490    4860 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-31.ec2.internal"
Oct 29 22:14:19.232202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232183    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.232270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232208    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-certificates\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.232270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232225    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7496f437-cb3c-4867-af37-16835f1134a1-config\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:19.232270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232240    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkw58\" (UniqueName: \"kubernetes.io/projected/7496f437-cb3c-4867-af37-16835f1134a1-kube-api-access-bkw58\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:19.232270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232257    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-multus-certs\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.232485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232304    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5pzj\" (UniqueName: \"kubernetes.io/projected/fb84513d-ca51-4920-bfc5-4240aa395064-kube-api-access-w5pzj\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:19.232485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232356    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r86gd\" (UniqueName: \"kubernetes.io/projected/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-kube-api-access-r86gd\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:19.232485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232428    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l65cw\" (UniqueName: \"kubernetes.io/projected/31bc5aff-808a-4d45-bcac-80d1bd626d47-kube-api-access-l65cw\") pod \"kas-connection-checker-685895c7c9-stxg2\" (UID: \"31bc5aff-808a-4d45-bcac-80d1bd626d47\") " pod="kube-system/kas-connection-checker-685895c7c9-stxg2"
Oct 29 22:14:19.232485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232438    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-certificates\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.232485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232463    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-tuned\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.232712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232490    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/529f43fd-d894-45ac-9143-2dc520293e16-cert\") pod \"ingress-canary-mld7c\" (UID: \"529f43fd-d894-45ac-9143-2dc520293e16\") " pod="openshift-ingress-canary/ingress-canary-mld7c"
Oct 29 22:14:19.232712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232514    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52gp6\" (UniqueName: \"kubernetes.io/projected/ccb78139-7ccc-4ac8-87bf-264d048488c4-kube-api-access-52gp6\") pod \"kas-connection-checker-685895c7c9-2z8f7\" (UID: \"ccb78139-7ccc-4ac8-87bf-264d048488c4\") " pod="kube-system/kas-connection-checker-685895c7c9-2z8f7"
Oct 29 22:14:19.232712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232536    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7496f437-cb3c-4867-af37-16835f1134a1-config\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:19.232712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232542    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntmn\" (UniqueName: \"kubernetes.io/projected/529f43fd-d894-45ac-9143-2dc520293e16-kube-api-access-vntmn\") pod \"ingress-canary-mld7c\" (UID: \"529f43fd-d894-45ac-9143-2dc520293e16\") " pod="openshift-ingress-canary/ingress-canary-mld7c"
Oct 29 22:14:19.232712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232547    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-tuned\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.232712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232569    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb84513d-ca51-4920-bfc5-4240aa395064-service-ca-bundle\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:19.232712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232594    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d56328b-9fa0-45cb-996f-65c1c1b8785c-dbus\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:19.232712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232620    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/20f39049-d7a1-46d8-8ebc-c0159906d689-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:19.232712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232663    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-installation-pull-secrets\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.232712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232695    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-etc-kubernetes\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.233021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232737    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f465b95-5af5-4d92-921b-24b0800934a0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:19.233021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232768    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e21cb68b-ed42-4009-9bf6-9700a98ffd42-iptables-alerter-script\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:14:19.233021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232795    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-sys-fs\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.233021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232844    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7099dc88-0238-49a5-b0fb-8d008e1b040e-config-volume\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:19.233021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232859    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-installation-pull-secrets\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.233021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232872    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5rt\" (UniqueName: \"kubernetes.io/projected/7099dc88-0238-49a5-b0fb-8d008e1b040e-kube-api-access-5d5rt\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:19.233021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232901    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f465b95-5af5-4d92-921b-24b0800934a0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:19.233021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232920    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7496f437-cb3c-4867-af37-16835f1134a1-trusted-ca\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:19.233021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232943    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-k8s-cni-cncf-io\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.233021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.232965    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-kubelet\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.233021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233007    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb84513d-ca51-4920-bfc5-4240aa395064-snapshots\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:19.233354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233031    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-stats-auth\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.233354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233051    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c49406-6e72-4bdd-8375-20dffea48c35-tmp-dir\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:14:19.233354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233168    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-node-log\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.233354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233193    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb84513d-ca51-4920-bfc5-4240aa395064-snapshots\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:19.233354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233209    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d56328b-9fa0-45cb-996f-65c1c1b8785c-kubelet-config\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:19.233354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233242    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7496f437-cb3c-4867-af37-16835f1134a1-trusted-ca\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:19.233354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233253    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9ac5e7-84bd-4b61-958b-f78cd316e348-crio-socket\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.233354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233254    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-stats-auth\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.233354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233271    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.233354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233240    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c49406-6e72-4bdd-8375-20dffea48c35-tmp-dir\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:14:19.233354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233290    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pkdx\" (UniqueName: \"kubernetes.io/projected/a342967b-704c-4947-92fa-e2dc475675a0-kube-api-access-6pkdx\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:19.233354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233343    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-cni-binary-copy\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233369    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233386    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972vc\" (UniqueName: \"kubernetes.io/projected/19e92d72-5f01-4c0f-9739-e094b74d5825-kube-api-access-972vc\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233403    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c8fff1-c08b-4437-944f-9ee33d186c42-konnectivity-ca\") pod \"konnectivity-agent-99lmd\" (UID: \"b6c8fff1-c08b-4437-944f-9ee33d186c42\") " pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233427    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-device-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233453    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7099dc88-0238-49a5-b0fb-8d008e1b040e-tmp-dir\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233478    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-daemon-config\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233519    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplzf\" (UniqueName: \"kubernetes.io/projected/6f6f3621-6d10-4afb-9438-c73cc3d4b13d-kube-api-access-dplzf\") pod \"volume-data-source-validator-58c9f5f46d-vmtz2\" (UID: \"6f6f3621-6d10-4afb-9438-c73cc3d4b13d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233533    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7099dc88-0238-49a5-b0fb-8d008e1b040e-tmp-dir\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233553    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovn-node-metrics-cert\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233579    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-system-cni-dir\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233608    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233641    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhfw\" (UniqueName: \"kubernetes.io/projected/0605dc25-2004-42f2-a856-2f0be8e75ca2-kube-api-access-hkhfw\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233668    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g29f\" (UniqueName: \"kubernetes.io/projected/8e0ef4de-c402-4c07-92f9-a7ec7d263113-kube-api-access-2g29f\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233686    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0da83403-18e0-4abd-8bbb-9323e80a3959-ca-trust-extracted\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.233703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233703    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgwps\" (UniqueName: \"kubernetes.io/projected/61824bf7-eca8-4554-8c54-81153bdcca82-kube-api-access-tgwps\") pod \"network-check-source-6f78d59f99-j4c22\" (UID: \"61824bf7-eca8-4554-8c54-81153bdcca82\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233727    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233744    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233761    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7drt\" (UniqueName: \"kubernetes.io/projected/93d782aa-ad96-4f81-a5a1-d356d022106b-kube-api-access-d7drt\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233779    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0da83403-18e0-4abd-8bbb-9323e80a3959-ca-trust-extracted\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233784    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-kubelet\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233873    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-cnibin\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233944    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-socket-dir-parent\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.233977    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f465b95-5af5-4d92-921b-24b0800934a0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234013    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c8fff1-c08b-4437-944f-9ee33d186c42-agent-certs\") pod \"konnectivity-agent-99lmd\" (UID: \"b6c8fff1-c08b-4437-944f-9ee33d186c42\") " pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234064    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234100    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-modprobe-d\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234122    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-slash\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234166    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f465b95-5af5-4d92-921b-24b0800934a0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234182    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234209    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234249    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9ac5e7-84bd-4b61-958b-f78cd316e348-data-volume\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234277    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5dd\" (UniqueName: \"kubernetes.io/projected/b822897d-559d-47b4-b9e8-9ba7f417a7bd-kube-api-access-8k5dd\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:19.234300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234299    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234311    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df9ac5e7-84bd-4b61-958b-f78cd316e348-data-volume\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234314    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234363    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-netns\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234417    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c8fff1-c08b-4437-944f-9ee33d186c42-agent-certs\") pod \"konnectivity-agent-99lmd\" (UID: \"b6c8fff1-c08b-4437-944f-9ee33d186c42\") " pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234459    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-run-ovn-kubernetes\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234489    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-var-lib-kubelet\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234547    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-systemd-units\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234583    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cvd\" (UniqueName: \"kubernetes.io/projected/eb9f947e-2e34-4b91-9e2b-1edafb918fef-kube-api-access-76cvd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234623    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbdg5\" (UniqueName: \"kubernetes.io/projected/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-api-access-hbdg5\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234649    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-default-certificate\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234673    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-trusted-ca\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234696    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-run\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234719    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-cni-bin\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234744    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e0ef4de-c402-4c07-92f9-a7ec7d263113-cni-binary-copy\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234803    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e21cb68b-ed42-4009-9bf6-9700a98ffd42-host-slash\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234841    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-cni-netd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234867    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-default-certificate\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234906    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-registration-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234922    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-hostroot\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234946    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qdlv\" (UniqueName: \"kubernetes.io/projected/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-kube-api-access-6qdlv\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.234981    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235040    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-sys\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.235058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235070    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-host\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235073    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-trusted-ca\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235089    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-tmp\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235115    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93d782aa-ad96-4f81-a5a1-d356d022106b-serviceca\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235155    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-tmp\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235180    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-system-cni-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235275    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-certificates\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235311    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-systemd\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235352    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93d782aa-ad96-4f81-a5a1-d356d022106b-serviceca\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235351    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-ovn\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235397    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-socket-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235427    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-certificates\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235460    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-image-registry-private-configuration\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235494    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a342967b-704c-4947-92fa-e2dc475675a0-serving-cert\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235522    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg246\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-kube-api-access-tg246\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235570    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysctl-d\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235594    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-cni-bin\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235626    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb84513d-ca51-4920-bfc5-4240aa395064-tmp\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235645    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-image-registry-private-configuration\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235655    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-bound-sa-token\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235660    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a342967b-704c-4947-92fa-e2dc475675a0-serving-cert\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235696    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxqn\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-kube-api-access-gwxqn\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235744    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4c49406-6e72-4bdd-8375-20dffea48c35-hosts-file\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235782    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb84513d-ca51-4920-bfc5-4240aa395064-tmp\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235815    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235860    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysconfig\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235894    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgg2\" (UniqueName: \"kubernetes.io/projected/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-kube-api-access-zjgg2\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235915    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93d782aa-ad96-4f81-a5a1-d356d022106b-host\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:14:19.235927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235932    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-installation-pull-secrets\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.235991    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-image-registry-private-configuration\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236066    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-systemd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236154    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-installation-pull-secrets\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236179    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-env-overrides\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236225    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-image-registry-private-configuration\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236230    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-cni-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236270    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57h7r\" (UniqueName: \"kubernetes.io/projected/9f465b95-5af5-4d92-921b-24b0800934a0-kube-api-access-57h7r\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236312    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rgp\" (UniqueName: \"kubernetes.io/projected/a4c49406-6e72-4bdd-8375-20dffea48c35-kube-api-access-68rgp\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236360    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljh6n\" (UniqueName: \"kubernetes.io/projected/1e6952c3-e47b-4c03-8804-04de63d8f1ee-kube-api-access-ljh6n\") pod \"kas-connection-checker-685895c7c9-6zmj5\" (UID: \"1e6952c3-e47b-4c03-8804-04de63d8f1ee\") " pod="kube-system/kas-connection-checker-685895c7c9-6zmj5"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236383    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-kubernetes\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236400    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mht74\" (UniqueName: \"kubernetes.io/projected/e21cb68b-ed42-4009-9bf6-9700a98ffd42-kube-api-access-mht74\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236419    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk9z5\" (UniqueName: \"kubernetes.io/projected/3611f6fc-6112-4b2e-8e22-75672c63c673-kube-api-access-qk9z5\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236435    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a342967b-704c-4947-92fa-e2dc475675a0-config\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236462    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovnkube-config\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236554    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovnkube-script-lib\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236587    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-etc-selinux\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236614    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-conf-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236644    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb84513d-ca51-4920-bfc5-4240aa395064-serving-cert\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236666    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0f94ef8-7c40-437b-a27e-36583be8bfba-ca-trust-extracted\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236682    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-cnibin\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236706    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236735    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236754    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0f94ef8-7c40-437b-a27e-36583be8bfba-ca-trust-extracted\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236780    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236791    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb84513d-ca51-4920-bfc5-4240aa395064-serving-cert\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236799    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-trusted-ca\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236839    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysctl-conf\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236872    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-etc-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236896    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236923    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.236989    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmzq\" (UniqueName: \"kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq\") pod \"network-check-target-sd9mh\" (UID: \"5345c4bc-ec8a-454b-83bf-363e0d1432e6\") " pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237018    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-lib-modules\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237032    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-run-netns\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237046    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-trusted-ca\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.237050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237051    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-var-lib-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.238190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237087    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.238190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237117    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7496f437-cb3c-4867-af37-16835f1134a1-serving-cert\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:19.238190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237168    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.238190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237194    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-bound-sa-token\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.238190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237209    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-cni-multus\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.238190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237221    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7496f437-cb3c-4867-af37-16835f1134a1-serving-cert\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:19.238190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237243    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb84513d-ca51-4920-bfc5-4240aa395064-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:19.238190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237378    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-log-socket\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.238190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.237671    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb84513d-ca51-4920-bfc5-4240aa395064-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:19.241237 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241219    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:14:19.241481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241423    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.241481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241438    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 29 22:14:19.241481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241442    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.241481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241423    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:14:19.241481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241470    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:14:19.241764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241488    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.241764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241513    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.241764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241708    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.241764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241721    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:14:19.241764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241742    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.241764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241750    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:14:19.241764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241760    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.242054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241804    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.242054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241841    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:14:19.242054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241842    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.242054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241871    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.242054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241879    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.242054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241890    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f584j\""
Oct 29 22:14:19.242054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.241915    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.242296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242155    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:14:19.242296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242180    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:14:19.242296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242180    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.242296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242193    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.242434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242412    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.242479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242470    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 29 22:14:19.242525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242485    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:14:19.242525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242487    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 29 22:14:19.242525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242499    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-977wj\""
Oct 29 22:14:19.242525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242501    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-d5ps2\""
Oct 29 22:14:19.242703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242482    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.242752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242717    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.242752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242742    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.242845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242755    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:14:19.242845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242758    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:14:19.242845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242807    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.242845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242812    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:14:19.242845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242826    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:14:19.242845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242837    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242869    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242870    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bwm9d\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242891    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kr68l\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242906    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cdnlg\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242914    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/529f43fd-d894-45ac-9143-2dc520293e16-cert\") pod \"ingress-canary-mld7c\" (UID: \"529f43fd-d894-45ac-9143-2dc520293e16\") " pod="openshift-ingress-canary/ingress-canary-mld7c"
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242927    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242952    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242978    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242990    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.242999    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243019    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243043    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xpwxr\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243045    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243056    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mwjl8\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243077    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243127    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7099dc88-0238-49a5-b0fb-8d008e1b040e-config-volume\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:19.243168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243148    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:14:19.243733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243205    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p5dkc\""
Oct 29 22:14:19.243733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243220    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e21cb68b-ed42-4009-9bf6-9700a98ffd42-iptables-alerter-script\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:14:19.243733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243270    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fz5pc\""
Oct 29 22:14:19.243733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243337    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8zjwp\""
Oct 29 22:14:19.243733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243375    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 29 22:14:19.243733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243526    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 29 22:14:19.243733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243640    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-cni-binary-copy\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.243733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243721    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c8fff1-c08b-4437-944f-9ee33d186c42-konnectivity-ca\") pod \"konnectivity-agent-99lmd\" (UID: \"b6c8fff1-c08b-4437-944f-9ee33d186c42\") " pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:14:19.244061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243744    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:19.244061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243783    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovn-node-metrics-cert\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.244061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243824    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-daemon-config\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.244061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.243925    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.245478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.245462    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e0ef4de-c402-4c07-92f9-a7ec7d263113-cni-binary-copy\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.246614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.246598    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-env-overrides\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.246692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.246617    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:19.246692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.246672    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovnkube-config\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.246801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.246714    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a342967b-704c-4947-92fa-e2dc475675a0-config\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:19.246857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.246824    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb9f947e-2e34-4b91-9e2b-1edafb918fef-ovnkube-script-lib\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.246994 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.246978    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3611f6fc-6112-4b2e-8e22-75672c63c673-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.247204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.247187    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d56328b-9fa0-45cb-996f-65c1c1b8785c-original-pull-secret\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:19.252842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.252825    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb84513d-ca51-4920-bfc5-4240aa395064-service-ca-bundle\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:19.252913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.252843    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/20f39049-d7a1-46d8-8ebc-c0159906d689-nginx-conf\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:19.253079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.253060    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52gp6\" (UniqueName: \"kubernetes.io/projected/ccb78139-7ccc-4ac8-87bf-264d048488c4-kube-api-access-52gp6\") pod \"kas-connection-checker-685895c7c9-2z8f7\" (UID: \"ccb78139-7ccc-4ac8-87bf-264d048488c4\") " pod="kube-system/kas-connection-checker-685895c7c9-2z8f7"
Oct 29 22:14:19.253391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.253370    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5pzj\" (UniqueName: \"kubernetes.io/projected/fb84513d-ca51-4920-bfc5-4240aa395064-kube-api-access-w5pzj\") pod \"insights-operator-7c4986d449-wfpts\" (UID: \"fb84513d-ca51-4920-bfc5-4240aa395064\") " pod="openshift-insights/insights-operator-7c4986d449-wfpts"
Oct 29 22:14:19.253606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.253589    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r86gd\" (UniqueName: \"kubernetes.io/projected/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-kube-api-access-r86gd\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:19.253653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.253607    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkw58\" (UniqueName: \"kubernetes.io/projected/7496f437-cb3c-4867-af37-16835f1134a1-kube-api-access-bkw58\") pod \"console-operator-7cd56b8dbc-7gx9p\" (UID: \"7496f437-cb3c-4867-af37-16835f1134a1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:19.253653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.253593    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntmn\" (UniqueName: \"kubernetes.io/projected/529f43fd-d894-45ac-9143-2dc520293e16-kube-api-access-vntmn\") pod \"ingress-canary-mld7c\" (UID: \"529f43fd-d894-45ac-9143-2dc520293e16\") " pod="openshift-ingress-canary/ingress-canary-mld7c"
Oct 29 22:14:19.253653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.253649    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-972vc\" (UniqueName: \"kubernetes.io/projected/19e92d72-5f01-4c0f-9739-e094b74d5825-kube-api-access-972vc\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.254533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.254513    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5rt\" (UniqueName: \"kubernetes.io/projected/7099dc88-0238-49a5-b0fb-8d008e1b040e-kube-api-access-5d5rt\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:19.254614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.254568    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l65cw\" (UniqueName: \"kubernetes.io/projected/31bc5aff-808a-4d45-bcac-80d1bd626d47-kube-api-access-l65cw\") pod \"kas-connection-checker-685895c7c9-stxg2\" (UID: \"31bc5aff-808a-4d45-bcac-80d1bd626d47\") " pod="kube-system/kas-connection-checker-685895c7c9-stxg2"
Oct 29 22:14:19.254855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.254839    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g29f\" (UniqueName: \"kubernetes.io/projected/8e0ef4de-c402-4c07-92f9-a7ec7d263113-kube-api-access-2g29f\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.255571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.255551    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qdlv\" (UniqueName: \"kubernetes.io/projected/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-kube-api-access-6qdlv\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:19.255852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.255832    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7drt\" (UniqueName: \"kubernetes.io/projected/93d782aa-ad96-4f81-a5a1-d356d022106b-kube-api-access-d7drt\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:14:19.255924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.255874    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplzf\" (UniqueName: \"kubernetes.io/projected/6f6f3621-6d10-4afb-9438-c73cc3d4b13d-kube-api-access-dplzf\") pod \"volume-data-source-validator-58c9f5f46d-vmtz2\" (UID: \"6f6f3621-6d10-4afb-9438-c73cc3d4b13d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vmtz2"
Oct 29 22:14:19.255958 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.255928    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhfw\" (UniqueName: \"kubernetes.io/projected/0605dc25-2004-42f2-a856-2f0be8e75ca2-kube-api-access-hkhfw\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.256048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.256029    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgwps\" (UniqueName: \"kubernetes.io/projected/61824bf7-eca8-4554-8c54-81153bdcca82-kube-api-access-tgwps\") pod \"network-check-source-6f78d59f99-j4c22\" (UID: \"61824bf7-eca8-4554-8c54-81153bdcca82\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j4c22"
Oct 29 22:14:19.256118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.256103    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pkdx\" (UniqueName: \"kubernetes.io/projected/a342967b-704c-4947-92fa-e2dc475675a0-kube-api-access-6pkdx\") pod \"service-ca-operator-544d749c8c-s2cf6\" (UID: \"a342967b-704c-4947-92fa-e2dc475675a0\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6"
Oct 29 22:14:19.257031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.257009    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbdg5\" (UniqueName: \"kubernetes.io/projected/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-api-access-hbdg5\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.257135 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.257099    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cvd\" (UniqueName: \"kubernetes.io/projected/eb9f947e-2e34-4b91-9e2b-1edafb918fef-kube-api-access-76cvd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.257509 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.257485    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk9z5\" (UniqueName: \"kubernetes.io/projected/3611f6fc-6112-4b2e-8e22-75672c63c673-kube-api-access-qk9z5\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.257981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.257956    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-bound-sa-token\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.258081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.257960    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57h7r\" (UniqueName: \"kubernetes.io/projected/9f465b95-5af5-4d92-921b-24b0800934a0-kube-api-access-57h7r\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg\" (UID: \"9f465b95-5af5-4d92-921b-24b0800934a0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg"
Oct 29 22:14:19.258201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.258080    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjgg2\" (UniqueName: \"kubernetes.io/projected/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-kube-api-access-zjgg2\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.258275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.258253    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxqn\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-kube-api-access-gwxqn\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.258502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.258481    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg246\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-kube-api-access-tg246\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.258551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.258538    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5dd\" (UniqueName: \"kubernetes.io/projected/b822897d-559d-47b4-b9e8-9ba7f417a7bd-kube-api-access-8k5dd\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:19.259832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.259813    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-bound-sa-token\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.260313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.260296    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mht74\" (UniqueName: \"kubernetes.io/projected/e21cb68b-ed42-4009-9bf6-9700a98ffd42-kube-api-access-mht74\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:14:19.260679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.260660    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljh6n\" (UniqueName: \"kubernetes.io/projected/1e6952c3-e47b-4c03-8804-04de63d8f1ee-kube-api-access-ljh6n\") pod \"kas-connection-checker-685895c7c9-6zmj5\" (UID: \"1e6952c3-e47b-4c03-8804-04de63d8f1ee\") " pod="kube-system/kas-connection-checker-685895c7c9-6zmj5"
Oct 29 22:14:19.260749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.260700    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rgp\" (UniqueName: \"kubernetes.io/projected/a4c49406-6e72-4bdd-8375-20dffea48c35-kube-api-access-68rgp\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:14:19.260873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.260854    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmzq\" (UniqueName: \"kubernetes.io/projected/5345c4bc-ec8a-454b-83bf-363e0d1432e6-kube-api-access-rrmzq\") pod \"network-check-target-sd9mh\" (UID: \"5345c4bc-ec8a-454b-83bf-363e0d1432e6\") " pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:19.338112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338087    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-device-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.338192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338148    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-system-cni-dir\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.338192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338168    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.338192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338182    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.338287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338196    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-kubelet\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.338287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338196    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-device-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.338287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338213    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-cnibin\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.338287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338260    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-kubelet\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.338287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338260    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-socket-dir-parent\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338282    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-system-cni-dir\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.338270    4860 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338339    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-cnibin\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338316    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.338370    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:19.838349044 +0000 UTC m=+1.863125171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : configmap references non-existent config key: service-ca.crt
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338383    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-socket-dir-parent\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338404    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-modprobe-d\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.338418    4860 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.338422    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls podName:df9ac5e7-84bd-4b61-958b-f78cd316e348 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:19.838404571 +0000 UTC m=+1.863180700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd9qn" (UID: "df9ac5e7-84bd-4b61-958b-f78cd316e348") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338463    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-slash\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338499    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338510    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-slash\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338526    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-modprobe-d\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.338550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338533    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.338547    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls podName:b822897d-559d-47b4-b9e8-9ba7f417a7bd nodeName:}" failed. No retries permitted until 2026-10-29 22:14:19.838535468 +0000 UTC m=+1.863311587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q9kr6" (UID: "b822897d-559d-47b4-b9e8-9ba7f417a7bd") : secret "samples-operator-tls" not found
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338563    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338577    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338623    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338643    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338658    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-netns\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338673    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-run-ovn-kubernetes\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338696    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-var-lib-kubelet\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.338706    4860 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338728    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-netns\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338737    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-run-ovn-kubernetes\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338744    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-systemd-units\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338768    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338771    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-var-lib-kubelet\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.338798    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert podName:20f39049-d7a1-46d8-8ebc-c0159906d689 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:19.838785627 +0000 UTC m=+1.863561737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9grl" (UID: "20f39049-d7a1-46d8-8ebc-c0159906d689") : secret "networking-console-plugin-cert" not found
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338806    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-systemd-units\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338830    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-run\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338875    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-cni-bin\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338882    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-run\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338900    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e21cb68b-ed42-4009-9bf6-9700a98ffd42-host-slash\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338918    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-cni-netd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338934    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-registration-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338938    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-cni-bin\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338979    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-cni-netd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338984    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e21cb68b-ed42-4009-9bf6-9700a98ffd42-host-slash\") pod \"iptables-alerter-7lbzl\" (UID: \"e21cb68b-ed42-4009-9bf6-9700a98ffd42\") " pod="openshift-network-operator/iptables-alerter-7lbzl"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.338979    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-hostroot\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339021    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-hostroot\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339062    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339071    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-registration-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.339092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339076    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-sys\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339107    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-host\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339129    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-system-cni-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339143    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-systemd\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339166    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-sys\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339176    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-host\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339186    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-system-cni-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.339187    4860 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.339203    4860 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8746f6cf-j8g6f: secret "image-registry-tls" not found
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339212    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-ovn\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339218    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-systemd\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339189    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-ovn\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.339269    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls podName:0da83403-18e0-4abd-8bbb-9323e80a3959 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:19.839259118 +0000 UTC m=+1.864035247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls") pod "image-registry-6f8746f6cf-j8g6f" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959") : secret "image-registry-tls" not found
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339291    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-socket-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339311    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysctl-d\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339487    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-cni-bin\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339508    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4c49406-6e72-4bdd-8375-20dffea48c35-hosts-file\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339520    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-socket-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339523    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysconfig\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339544    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93d782aa-ad96-4f81-a5a1-d356d022106b-host\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339559    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-systemd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339522    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-cni-bin\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339573    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-cni-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339593    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysctl-d\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339633    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4c49406-6e72-4bdd-8375-20dffea48c35-hosts-file\") pod \"node-resolver-m7gqm\" (UID: \"a4c49406-6e72-4bdd-8375-20dffea48c35\") " pod="openshift-dns/node-resolver-m7gqm"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339620    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93d782aa-ad96-4f81-a5a1-d356d022106b-host\") pod \"node-ca-x8qdl\" (UID: \"93d782aa-ad96-4f81-a5a1-d356d022106b\") " pod="openshift-image-registry/node-ca-x8qdl"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339637    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-run-systemd\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339655    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-kubernetes\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339660    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysconfig\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339619    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-cni-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339697    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-kubernetes\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339748    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-etc-selinux\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339778    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-conf-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339808    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-cnibin\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339815    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-multus-conf-dir\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339814    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-etc-selinux\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339838    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3611f6fc-6112-4b2e-8e22-75672c63c673-cnibin\") pod \"multus-additional-cni-plugins-xqtzn\" (UID: \"3611f6fc-6112-4b2e-8e22-75672c63c673\") " pod="openshift-multus/multus-additional-cni-plugins-xqtzn"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339843    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339876    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.339891    4860 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339899    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysctl-conf\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.339929    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs podName:a97d96fb-da2b-4064-b6bb-ac7f3cf2270b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:19.839917664 +0000 UTC m=+1.864693786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs") pod "network-metrics-daemon-l9rds" (UID: "a97d96fb-da2b-4064-b6bb-ac7f3cf2270b") : secret "metrics-daemon-secret" not found
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339942    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-etc-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339958    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.339965    4860 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339972    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-lib-modules\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.339996    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-run-netns\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340042    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-etc-sysctl-conf\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.340058    4860 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340064    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-etc-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340088    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-var-lib-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340049    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-host-run-netns\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.340118    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls podName:7099dc88-0238-49a5-b0fb-8d008e1b040e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:19.840101614 +0000 UTC m=+1.864877724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls") pod "dns-default-5sn5b" (UID: "7099dc88-0238-49a5-b0fb-8d008e1b040e") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340114    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c23cb73-3b67-46d7-94ab-ad2c0d9473de-lib-modules\") pod \"tuned-mmk2r\" (UID: \"5c23cb73-3b67-46d7-94ab-ad2c0d9473de\") " pod="openshift-cluster-node-tuning-operator/tuned-mmk2r"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340128    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-var-lib-openvswitch\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.340131    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls podName:f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d nodeName:}" failed. No retries permitted until 2026-10-29 22:14:19.840125247 +0000 UTC m=+1.864901355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n98ww" (UID: "f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d") : secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:19.340139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340155    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340191    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340189    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340257    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-cni-multus\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340284    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-log-socket\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340300    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-cni-multus\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340306    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340367    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-log-socket\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340390    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-multus-certs\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.340423    4860 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.340436    4860 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9f64d5bbb-8pnfd: secret "image-registry-tls" not found
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340460    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-multus-certs\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340434    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d56328b-9fa0-45cb-996f-65c1c1b8785c-dbus\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340482    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d56328b-9fa0-45cb-996f-65c1c1b8785c-dbus\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.340489    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls podName:d0f94ef8-7c40-437b-a27e-36583be8bfba nodeName:}" failed. No retries permitted until 2026-10-29 22:14:19.840473708 +0000 UTC m=+1.865249820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls") pod "image-registry-9f64d5bbb-8pnfd" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba") : secret "image-registry-tls" not found
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340528    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-etc-kubernetes\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340554    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-sys-fs\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340603    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-etc-kubernetes\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340609    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-k8s-cni-cncf-io\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340656    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-kubelet\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340686    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-node-log\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340695    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0605dc25-2004-42f2-a856-2f0be8e75ca2-sys-fs\") pod \"aws-ebs-csi-driver-node-vmwr5\" (UID: \"0605dc25-2004-42f2-a856-2f0be8e75ca2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmwr5"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340656    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-run-k8s-cni-cncf-io\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340708    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d56328b-9fa0-45cb-996f-65c1c1b8785c-kubelet-config\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340745    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9ac5e7-84bd-4b61-958b-f78cd316e348-crio-socket\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340757    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb9f947e-2e34-4b91-9e2b-1edafb918fef-node-log\") pod \"ovnkube-node-774ht\" (UID: \"eb9f947e-2e34-4b91-9e2b-1edafb918fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340761    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e0ef4de-c402-4c07-92f9-a7ec7d263113-host-var-lib-kubelet\") pod \"multus-75pjw\" (UID: \"8e0ef4de-c402-4c07-92f9-a7ec7d263113\") " pod="openshift-multus/multus-75pjw"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340759    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d56328b-9fa0-45cb-996f-65c1c1b8785c-kubelet-config\") pod \"global-pull-secret-syncer-stxld\" (UID: \"7d56328b-9fa0-45cb-996f-65c1c1b8785c\") " pod="kube-system/global-pull-secret-syncer-stxld"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340769    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.340842    4860 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.340864    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df9ac5e7-84bd-4b61-958b-f78cd316e348-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.340914    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:19.840898392 +0000 UTC m=+1.865674500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : secret "router-metrics-certs-default" not found
Oct 29 22:14:19.342478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.341313    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df9ac5e7-84bd-4b61-958b-f78cd316e348-crio-socket\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.375134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.375105    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:19.427458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.427441    4860 scope.go:122] "RemoveContainer" containerID="7c727fc6b491e3c7a5d226c6cccc83981ca0988ffe7eb5d016e5d71adef13b31"
Oct 29 22:14:19.845526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.845464    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:19.845526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.845492    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:19.845671 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.845529    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:19.845671 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845605    4860 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:19.845671 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.845603    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:19.845671 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845611    4860 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 22:14:19.845671 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845643    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls podName:7099dc88-0238-49a5-b0fb-8d008e1b040e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:20.84563293 +0000 UTC m=+2.870409039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls") pod "dns-default-5sn5b" (UID: "7099dc88-0238-49a5-b0fb-8d008e1b040e") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:19.845671 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845645    4860 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:19.845671 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845676    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs podName:a97d96fb-da2b-4064-b6bb-ac7f3cf2270b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:20.845661789 +0000 UTC m=+2.870437915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs") pod "network-metrics-daemon-l9rds" (UID: "a97d96fb-da2b-4064-b6bb-ac7f3cf2270b") : secret "metrics-daemon-secret" not found
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.845696    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845724    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls podName:f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d nodeName:}" failed. No retries permitted until 2026-10-29 22:14:20.845714792 +0000 UTC m=+2.870490904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n98ww" (UID: "f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d") : secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.845749    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845758    4860 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.845767    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.845792    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845812    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:20.845801864 +0000 UTC m=+2.870577974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : secret "router-metrics-certs-default" not found
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845831    4860 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845840    4860 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845765    4860 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.845861    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845876    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls podName:b822897d-559d-47b4-b9e8-9ba7f417a7bd nodeName:}" failed. No retries permitted until 2026-10-29 22:14:20.845865932 +0000 UTC m=+2.870642055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q9kr6" (UID: "b822897d-559d-47b4-b9e8-9ba7f417a7bd") : secret "samples-operator-tls" not found
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845883    4860 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9f64d5bbb-8pnfd: secret "image-registry-tls" not found
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845892    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls podName:df9ac5e7-84bd-4b61-958b-f78cd316e348 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:20.845883548 +0000 UTC m=+2.870659659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd9qn" (UID: "df9ac5e7-84bd-4b61-958b-f78cd316e348") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845911    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:20.845900744 +0000 UTC m=+2.870676859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : configmap references non-existent config key: service-ca.crt
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845924    4860 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845939    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls podName:d0f94ef8-7c40-437b-a27e-36583be8bfba nodeName:}" failed. No retries permitted until 2026-10-29 22:14:20.845924047 +0000 UTC m=+2.870700186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls") pod "image-registry-9f64d5bbb-8pnfd" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba") : secret "image-registry-tls" not found
Oct 29 22:14:19.845986 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.845962    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert podName:20f39049-d7a1-46d8-8ebc-c0159906d689 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:20.845951731 +0000 UTC m=+2.870727840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-w9grl" (UID: "20f39049-d7a1-46d8-8ebc-c0159906d689") : secret "networking-console-plugin-cert" not found
Oct 29 22:14:19.846525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:19.846014    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:19.846525 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.846106    4860 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 22:14:19.846525 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.846115    4860 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8746f6cf-j8g6f: secret "image-registry-tls" not found
Oct 29 22:14:19.846525 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:19.846142    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls podName:0da83403-18e0-4abd-8bbb-9323e80a3959 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:20.84613548 +0000 UTC m=+2.870911616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls") pod "image-registry-6f8746f6cf-j8g6f" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959") : secret "image-registry-tls" not found
Oct 29 22:14:20.168910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.168845    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:14:20.169348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.169288    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/1.log"
Oct 29 22:14:20.169410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.169339    4860 generic.go:356] "Generic (PLEG): container finished" podID="7496f437-cb3c-4867-af37-16835f1134a1" containerID="234323f7a7e534ef47a04903748f65690f10f6bb827c0a83df4c6d6ffeb15849" exitCode=255
Oct 29 22:14:20.169410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.169354    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p" event={"ID":"7496f437-cb3c-4867-af37-16835f1134a1","Type":"ContainerDied","Data":"234323f7a7e534ef47a04903748f65690f10f6bb827c0a83df4c6d6ffeb15849"}
Oct 29 22:14:20.169513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.169412    4860 scope.go:122] "RemoveContainer" containerID="7c727fc6b491e3c7a5d226c6cccc83981ca0988ffe7eb5d016e5d71adef13b31"
Oct 29 22:14:20.169996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.169976    4860 scope.go:122] "RemoveContainer" containerID="234323f7a7e534ef47a04903748f65690f10f6bb827c0a83df4c6d6ffeb15849"
Oct 29 22:14:20.170170 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.170149    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7gx9p_openshift-console-operator(7496f437-cb3c-4867-af37-16835f1134a1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p" podUID="7496f437-cb3c-4867-af37-16835f1134a1"
Oct 29 22:14:20.291211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.291182    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:20.470164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.470138    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:20.584956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.584929    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:20.857007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.856922    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:20.857007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.856967    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:20.857007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.856987    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:20.857229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.857014    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:20.857229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.857049    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:20.857229 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857075    4860 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 22:14:20.857229 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857098    4860 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:20.857229 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857119    4860 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:20.857229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.857141    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:20.857229 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857161    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls podName:f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.857143007 +0000 UTC m=+4.881919150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n98ww" (UID: "f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d") : secret "cluster-monitoring-operator-tls" not found
Oct 29 22:14:20.857229 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857189    4860 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 22:14:20.857229 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857197    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls podName:7099dc88-0238-49a5-b0fb-8d008e1b040e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.857185223 +0000 UTC m=+4.881961346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls") pod "dns-default-5sn5b" (UID: "7099dc88-0238-49a5-b0fb-8d008e1b040e") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:20.857229 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857213    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs podName:a97d96fb-da2b-4064-b6bb-ac7f3cf2270b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.857204672 +0000 UTC m=+4.881980788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs") pod "network-metrics-daemon-l9rds" (UID: "a97d96fb-da2b-4064-b6bb-ac7f3cf2270b") : secret "metrics-daemon-secret" not found
Oct 29 22:14:20.857229 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857221    4860 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:20.857229 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857227    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.857219302 +0000 UTC m=+4.881995412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : secret "router-metrics-certs-default" not found
Oct 29 22:14:20.857726 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857262    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls podName:df9ac5e7-84bd-4b61-958b-f78cd316e348 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.857248222 +0000 UTC m=+4.882024345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd9qn" (UID: "df9ac5e7-84bd-4b61-958b-f78cd316e348") : secret "insights-runtime-extractor-tls" not found
Oct 29 22:14:20.857726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.857298    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:20.857726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.857360    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:20.857726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.857381    4860 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:14:20.857726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.857397    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:20.857726 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857420    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.857397365 +0000 UTC m=+4.882173474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : configmap references non-existent config key: service-ca.crt
Oct 29 22:14:20.857726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.857471    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:20.857726 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857493    4860 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 22:14:20.857726 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:20.857548    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls podName:b822897d-559d-47b4-b9e8-9ba7f417a7bd nodeName:}" failed. No retries permitted until 2026-10-29 22:14:22.857528525 +0000 UTC m=+4.882304636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q9kr6" (UID: "b822897d-559d-47b4-b9e8-9ba7f417a7bd") : secret "samples-operator-tls" not found
Oct 29 22:14:20.859744 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.859720    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/20f39049-d7a1-46d8-8ebc-c0159906d689-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-w9grl\" (UID: \"20f39049-d7a1-46d8-8ebc-c0159906d689\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:20.859884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.859864    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") pod \"image-registry-6f8746f6cf-j8g6f\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") " pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:20.859933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.859868    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") pod \"image-registry-9f64d5bbb-8pnfd\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") " pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:20.928300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.928270    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:20.930422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.930403    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl"
Oct 29 22:14:20.936101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:20.936084    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:21.071095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:21.071066    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:21.174591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:21.174542    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:14:21.255319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:21.255300    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-w9grl"]
Oct 29 22:14:21.255866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:21.255841    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f39049_d7a1_46d8_8ebc_c0159906d689.slice/crio-c8f6deb7254291c9eb847c5cc104faab389fcec1e1f0014162b5c19f3094257d WatchSource:0}: Error finding container c8f6deb7254291c9eb847c5cc104faab389fcec1e1f0014162b5c19f3094257d: Status 404 returned error can't find the container with id c8f6deb7254291c9eb847c5cc104faab389fcec1e1f0014162b5c19f3094257d
Oct 29 22:14:21.259367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:21.258646    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f8746f6cf-j8g6f"]
Oct 29 22:14:21.259367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:21.259025    4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:14:21.268438 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:21.268415    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da83403_18e0_4abd_8bbb_9323e80a3959.slice/crio-584141ef3b8938ec229fa962a0d394109dbcbc8b1896aadc081a03f92df85cbc WatchSource:0}: Error finding container 584141ef3b8938ec229fa962a0d394109dbcbc8b1896aadc081a03f92df85cbc: Status 404 returned error can't find the container with id 584141ef3b8938ec229fa962a0d394109dbcbc8b1896aadc081a03f92df85cbc
Oct 29 22:14:21.269609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:21.269591    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9f64d5bbb-8pnfd"]
Oct 29 22:14:21.270348 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:21.270311    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f94ef8_7c40_437b_a27e_36583be8bfba.slice/crio-d212502e014b15160dee880b88cb738266d4c04e2e4d3a9a06358edc53432175 WatchSource:0}: Error finding container d212502e014b15160dee880b88cb738266d4c04e2e4d3a9a06358edc53432175: Status 404 returned error can't find the container with id d212502e014b15160dee880b88cb738266d4c04e2e4d3a9a06358edc53432175
Oct 29 22:14:21.436027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:21.435999    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:21.578597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:21.578565    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:21.848175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:21.848081    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:22.040261 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.040200    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:22.072031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.072010    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m7gqm_a4c49406-6e72-4bdd-8375-20dffea48c35/dns-node-resolver/0.log"
Oct 29 22:14:22.181387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.181354    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f" event={"ID":"0da83403-18e0-4abd-8bbb-9323e80a3959","Type":"ContainerStarted","Data":"208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea"}
Oct 29 22:14:22.181832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.181419    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f" event={"ID":"0da83403-18e0-4abd-8bbb-9323e80a3959","Type":"ContainerStarted","Data":"584141ef3b8938ec229fa962a0d394109dbcbc8b1896aadc081a03f92df85cbc"}
Oct 29 22:14:22.182712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.182688    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl" event={"ID":"20f39049-d7a1-46d8-8ebc-c0159906d689","Type":"ContainerStarted","Data":"c8f6deb7254291c9eb847c5cc104faab389fcec1e1f0014162b5c19f3094257d"}
Oct 29 22:14:22.184489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.184457    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd" event={"ID":"d0f94ef8-7c40-437b-a27e-36583be8bfba","Type":"ContainerStarted","Data":"9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c"}
Oct 29 22:14:22.184599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.184504    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd" event={"ID":"d0f94ef8-7c40-437b-a27e-36583be8bfba","Type":"ContainerStarted","Data":"d212502e014b15160dee880b88cb738266d4c04e2e4d3a9a06358edc53432175"}
Oct 29 22:14:22.238977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.238921    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:22.274353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.273994    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6f8746f6cf-j8g6f_0da83403-18e0-4abd-8bbb-9323e80a3959/registry/0.log"
Oct 29 22:14:22.284232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.284209    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:22.448045 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.447998    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:22.471680 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.471659    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-9f64d5bbb-8pnfd_d0f94ef8-7c40-437b-a27e-36583be8bfba/registry/0.log"
Oct 29 22:14:22.685180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.685156    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:22.877804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.877746    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:22.877804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.877783    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:22.877955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.877807    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:22.877955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.877835    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:22.877955 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:22.877893    4860 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 22:14:22.877955 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:22.877917    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:26.877899478 +0000 UTC m=+8.902675588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : configmap references non-existent config key: service-ca.crt
Oct 29 22:14:22.877955 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:22.877943    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:26.877931564 +0000 UTC m=+8.902707686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : secret "router-metrics-certs-default" not found
Oct 29 22:14:22.878203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.877958    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:22.878203 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:22.877977    4860 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 22:14:22.878203 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:22.878032    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls podName:b822897d-559d-47b4-b9e8-9ba7f417a7bd nodeName:}" failed. No retries permitted until 2026-10-29 22:14:26.878015383 +0000 UTC m=+8.902791507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-q9kr6" (UID: "b822897d-559d-47b4-b9e8-9ba7f417a7bd") : secret "samples-operator-tls" not found
Oct 29 22:14:22.878203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.877980    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:22.878203 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:22.878054    4860 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 22:14:22.878203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.878085    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:22.878203 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:22.878095    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs podName:a97d96fb-da2b-4064-b6bb-ac7f3cf2270b nodeName:}" failed. No retries permitted until 2026-10-29 22:14:26.878082916 +0000 UTC m=+8.902859027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs") pod "network-metrics-daemon-l9rds" (UID: "a97d96fb-da2b-4064-b6bb-ac7f3cf2270b") : secret "metrics-daemon-secret" not found
Oct 29 22:14:22.878203 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:22.878145    4860 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 22:14:22.878203 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:22.878181    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls podName:7099dc88-0238-49a5-b0fb-8d008e1b040e nodeName:}" failed. No retries permitted until 2026-10-29 22:14:26.878169067 +0000 UTC m=+8.902945193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls") pod "dns-default-5sn5b" (UID: "7099dc88-0238-49a5-b0fb-8d008e1b040e") : secret "dns-default-metrics-tls" not found
Oct 29 22:14:22.880182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.880161    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df9ac5e7-84bd-4b61-958b-f78cd316e348-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd9qn\" (UID: \"df9ac5e7-84bd-4b61-958b-f78cd316e348\") " pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:22.880257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:22.880243    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n98ww\" (UID: \"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:23.029547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:23.029518    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"
Oct 29 22:14:23.038177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:23.038162    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pd9qn"
Oct 29 22:14:23.072343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:23.072294    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x8qdl_93d782aa-ad96-4f81-a5a1-d356d022106b/node-ca/0.log"
Oct 29 22:14:23.188911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:23.188886    4860 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:14:23.188911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:23.188886    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-w9grl" event={"ID":"20f39049-d7a1-46d8-8ebc-c0159906d689","Type":"ContainerStarted","Data":"f89221863e8d162439b2c7286678801e04e2e16e9a82710233a1dabb796474f4"}
Oct 29 22:14:23.189388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:23.189070    4860 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:14:23.324298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:23.324274    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww"]
Oct 29 22:14:23.324543 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:23.324521    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1afad65_7b45_46ac_b42d_d4cf7a5b8e8d.slice/crio-79212e46ffe50e4b9f7b68b63d2823ffc18dc34d5cd1614c7e3c3c43164c6d06 WatchSource:0}: Error finding container 79212e46ffe50e4b9f7b68b63d2823ffc18dc34d5cd1614c7e3c3c43164c6d06: Status 404 returned error can't find the container with id 79212e46ffe50e4b9f7b68b63d2823ffc18dc34d5cd1614c7e3c3c43164c6d06
Oct 29 22:14:23.357691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:23.357670    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pd9qn"]
Oct 29 22:14:23.358371 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:23.358347    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9ac5e7_84bd_4b61_958b_f78cd316e348.slice/crio-0634a0f885a565b844d27780f233743e4531eabfe1ed94f213a1335549ef01c0 WatchSource:0}: Error finding container 0634a0f885a565b844d27780f233743e4531eabfe1ed94f213a1335549ef01c0: Status 404 returned error can't find the container with id 0634a0f885a565b844d27780f233743e4531eabfe1ed94f213a1335549ef01c0
Oct 29 22:14:23.642736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:23.642677    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:23.671870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:23.671848    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mld7c_529f43fd-d894-45ac-9143-2dc520293e16/serve-healthcheck-canary/0.log"
Oct 29 22:14:23.795111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:23.795088    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:23.844844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:23.844816    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:24.038555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:24.038527    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:24.136024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:24.135998    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:24.136245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:24.136232    4860 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:14:24.195163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:24.195123    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd9qn" event={"ID":"df9ac5e7-84bd-4b61-958b-f78cd316e348","Type":"ContainerStarted","Data":"dc90671053942cd5d2e0490d35f3cc3801b342ea8f96227db29c100957f8f882"}
Oct 29 22:14:24.195558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:24.195227    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd9qn" event={"ID":"df9ac5e7-84bd-4b61-958b-f78cd316e348","Type":"ContainerStarted","Data":"e36b8571ad99db84f7ac8396a0b50408e0c4941dd0107aa5e668969e062de9b3"}
Oct 29 22:14:24.195558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:24.195258    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd9qn" event={"ID":"df9ac5e7-84bd-4b61-958b-f78cd316e348","Type":"ContainerStarted","Data":"0634a0f885a565b844d27780f233743e4531eabfe1ed94f213a1335549ef01c0"}
Oct 29 22:14:24.196670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:24.196645    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww" event={"ID":"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d","Type":"ContainerStarted","Data":"79212e46ffe50e4b9f7b68b63d2823ffc18dc34d5cd1614c7e3c3c43164c6d06"}
Oct 29 22:14:24.237722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:24.237656    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:24.443566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:24.443539    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:24.473286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:24.473240    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg_9f465b95-5af5-4d92-921b-24b0800934a0/kube-storage-version-migrator-operator/0.log"
Oct 29 22:14:24.686188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:24.686157    4860 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:24.686643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:24.686598    4860 scope.go:122] "RemoveContainer" containerID="234323f7a7e534ef47a04903748f65690f10f6bb827c0a83df4c6d6ffeb15849"
Oct 29 22:14:24.686863 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:24.686839    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7gx9p_openshift-console-operator(7496f437-cb3c-4867-af37-16835f1134a1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p" podUID="7496f437-cb3c-4867-af37-16835f1134a1"
Oct 29 22:14:25.202019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:25.201950    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n98ww" event={"ID":"f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d","Type":"ContainerStarted","Data":"89fc3d653010043fa176925ffeb081d04d3c45af795081e2a25c06cbc88b3f46"}
Oct 29 22:14:25.555676 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:25.555645    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:25.674585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:25.674527    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:25.841746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:25.841721    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:25.947338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:25.947300    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:26.069648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.069622    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:26.206970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.206893    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd9qn" event={"ID":"df9ac5e7-84bd-4b61-958b-f78cd316e348","Type":"ContainerStarted","Data":"8b5d608b2bb99f97689a37353dab4face1e28788d928a916fd588a9638367740"}
Oct 29 22:14:26.224939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.224917    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:26.231201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.231180    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:26.443518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.443464    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:26.913768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.913737    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:26.913925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.913773    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:26.913925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.913795    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:26.913925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.913839    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:26.913925 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:26.913913    4860 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 22:14:26.914134 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:26.913919    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:34.913902806 +0000 UTC m=+16.938678926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : configmap references non-existent config key: service-ca.crt
Oct 29 22:14:26.914134 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:26.913989    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs podName:19e92d72-5f01-4c0f-9739-e094b74d5825 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:34.91397101 +0000 UTC m=+16.938747126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs") pod "router-default-7b6b64f55f-9l4xp" (UID: "19e92d72-5f01-4c0f-9739-e094b74d5825") : secret "router-metrics-certs-default" not found
Oct 29 22:14:26.914134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.914019    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:26.916272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.916249    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7099dc88-0238-49a5-b0fb-8d008e1b040e-metrics-tls\") pod \"dns-default-5sn5b\" (UID: \"7099dc88-0238-49a5-b0fb-8d008e1b040e\") " pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:26.916375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.916301    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a97d96fb-da2b-4064-b6bb-ac7f3cf2270b-metrics-certs\") pod \"network-metrics-daemon-l9rds\" (UID: \"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b\") " pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:26.916427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.916417    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b822897d-559d-47b4-b9e8-9ba7f417a7bd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-q9kr6\" (UID: \"b822897d-559d-47b4-b9e8-9ba7f417a7bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:26.928359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.928343    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"
Oct 29 22:14:26.933603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.933583    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9rds"
Oct 29 22:14:26.936498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.936480    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:26.940750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:26.940728    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:27.053897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:27.053872    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:27.234046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:27.234026    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6"]
Oct 29 22:14:27.242124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:27.242102    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l9rds"]
Oct 29 22:14:27.245795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:27.245522    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5sn5b"]
Oct 29 22:14:27.246548 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:27.246520    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97d96fb_da2b_4064_b6bb_ac7f3cf2270b.slice/crio-837256ae05126757a4966d99e8cd4c8c0d311d9b55a0352ddf2fc85ff2cd01fb WatchSource:0}: Error finding container 837256ae05126757a4966d99e8cd4c8c0d311d9b55a0352ddf2fc85ff2cd01fb: Status 404 returned error can't find the container with id 837256ae05126757a4966d99e8cd4c8c0d311d9b55a0352ddf2fc85ff2cd01fb
Oct 29 22:14:27.246776 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:27.246755    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7099dc88_0238_49a5_b0fb_8d008e1b040e.slice/crio-43ae98883f1645f538843a2ddfb14fa3740e4770e9ea2a9f5a182712c145a081 WatchSource:0}: Error finding container 43ae98883f1645f538843a2ddfb14fa3740e4770e9ea2a9f5a182712c145a081: Status 404 returned error can't find the container with id 43ae98883f1645f538843a2ddfb14fa3740e4770e9ea2a9f5a182712c145a081
Oct 29 22:14:27.637019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:27.636939    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:27.834187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:27.834153    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:28.032771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:28.032730    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:28.215106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:28.215068    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5sn5b" event={"ID":"7099dc88-0238-49a5-b0fb-8d008e1b040e","Type":"ContainerStarted","Data":"43ae98883f1645f538843a2ddfb14fa3740e4770e9ea2a9f5a182712c145a081"}
Oct 29 22:14:28.216920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:28.216886    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9rds" event={"ID":"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b","Type":"ContainerStarted","Data":"837256ae05126757a4966d99e8cd4c8c0d311d9b55a0352ddf2fc85ff2cd01fb"}
Oct 29 22:14:28.219478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:28.219451    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6" event={"ID":"b822897d-559d-47b4-b9e8-9ba7f417a7bd","Type":"ContainerStarted","Data":"c87f53c6eb0b11d80e09244f212993d9bca0f2c0b4ea69e7bed8954ce30c538f"}
Oct 29 22:14:28.239033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:28.239006    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:28.356974 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:28.356950    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:28.468715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:28.468682    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:28.584789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:28.584707    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:28.585173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:28.585158    4860 scope.go:122] "RemoveContainer" containerID="234323f7a7e534ef47a04903748f65690f10f6bb827c0a83df4c6d6ffeb15849"
Oct 29 22:14:28.585381 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:28.585356    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7gx9p_openshift-console-operator(7496f437-cb3c-4867-af37-16835f1134a1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p" podUID="7496f437-cb3c-4867-af37-16835f1134a1"
Oct 29 22:14:29.226970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:29.226924    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9rds" event={"ID":"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b","Type":"ContainerStarted","Data":"71d96774d726635bc267b8b163970c6dd63c5e11035a664c428e80f073e436b7"}
Oct 29 22:14:29.227073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:29.227004    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9rds" event={"ID":"a97d96fb-da2b-4064-b6bb-ac7f3cf2270b","Type":"ContainerStarted","Data":"be1dc7dd6ab2411457e090b8f611a8511765c1d180dfdd336f39b11a0b0ab1b9"}
Oct 29 22:14:29.228822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:29.228800    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6" event={"ID":"b822897d-559d-47b4-b9e8-9ba7f417a7bd","Type":"ContainerStarted","Data":"d6ae273a51b2b3f837c9c6e21c204e8459976e43d365fb74704042651f98234a"}
Oct 29 22:14:29.230701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:29.230681    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5sn5b" event={"ID":"7099dc88-0238-49a5-b0fb-8d008e1b040e","Type":"ContainerStarted","Data":"7ba44d4846decc58c0a7787bf1f727f1ddfca55c86bace5e0f3e3322dd2116c6"}
Oct 29 22:14:29.230868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:29.230855    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:29.638500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:29.638426    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:29.797752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:29.797716    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:29.910719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:29.910662    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:30.038685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.038651    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:30.104588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.104566    4860 scope.go:122] "RemoveContainer" containerID="234323f7a7e534ef47a04903748f65690f10f6bb827c0a83df4c6d6ffeb15849"
Oct 29 22:14:30.237343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.236313    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:30.240611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.240578    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5sn5b" event={"ID":"7099dc88-0238-49a5-b0fb-8d008e1b040e","Type":"ContainerStarted","Data":"4636f136d652093cf71fb5f817b85760d39fbacef553fb1ddf0ca0ee662c13d2"}
Oct 29 22:14:30.242913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.242896    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:14:30.243037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.242985    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p" event={"ID":"7496f437-cb3c-4867-af37-16835f1134a1","Type":"ContainerStarted","Data":"fa24995d8b2c28980416ce8efa05890c9ebfb0ef191e26a1021c368eebc8db8e"}
Oct 29 22:14:30.243306 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.243289    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:30.244282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.244260    4860 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-7gx9p container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" start-of-body=
Oct 29 22:14:30.244387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.244305    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p" podUID="7496f437-cb3c-4867-af37-16835f1134a1" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused"
Oct 29 22:14:30.244964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.244942    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-q9kr6" event={"ID":"b822897d-559d-47b4-b9e8-9ba7f417a7bd","Type":"ContainerStarted","Data":"e3d35f65fefdf47eb51b10c449e51afe066732bb5e34e643e2b1e88b217fdda2"}
Oct 29 22:14:30.429700 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.429671    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:30.846162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.846129    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:30.963674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:30.963645    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:31.077960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:31.077930    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:31.187475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:31.187429    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:31.252651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:31.252629    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7gx9p"
Oct 29 22:14:31.308150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:31.308127    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:31.414572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:31.414552    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:31.634112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:31.634055    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:32.202893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:32.202867    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:32.310207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:32.310180    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:34.978069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:34.978038    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:34.978586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:34.978085    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:34.978803 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:34.978780    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19e92d72-5f01-4c0f-9739-e094b74d5825-service-ca-bundle\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:34.980401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:34.980377    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19e92d72-5f01-4c0f-9739-e094b74d5825-metrics-certs\") pod \"router-default-7b6b64f55f-9l4xp\" (UID: \"19e92d72-5f01-4c0f-9739-e094b74d5825\") " pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:35.027280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:35.027253    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:35.211318 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:35.211289    4860 patch_prober.go:29] interesting pod/image-registry-6f8746f6cf-j8g6f container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 29 22:14:35.211468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:35.211354    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f" podUID="0da83403-18e0-4abd-8bbb-9323e80a3959" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:35.332410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:35.332384    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7b6b64f55f-9l4xp"]
Oct 29 22:14:35.333527 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:35.333506    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e92d72_5f01_4c0f_9739_e094b74d5825.slice/crio-abc5735b5fa7eb315dae76e7759be72eb4e632e536a05b213fcf968cff4eb52c WatchSource:0}: Error finding container abc5735b5fa7eb315dae76e7759be72eb4e632e536a05b213fcf968cff4eb52c: Status 404 returned error can't find the container with id abc5735b5fa7eb315dae76e7759be72eb4e632e536a05b213fcf968cff4eb52c
Oct 29 22:14:35.646514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:35.646441    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9f64d5bbb-8pnfd"]
Oct 29 22:14:35.779030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:35.778965    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:35.904294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:35.904223    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:35.972461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:35.972439    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-774ht"
Oct 29 22:14:36.020914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:36.020890    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:36.266142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:36.266111    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b6b64f55f-9l4xp" event={"ID":"19e92d72-5f01-4c0f-9739-e094b74d5825","Type":"ContainerStarted","Data":"e5e7888baeffaf3d43964a9a04eca280a8c005dd98ca5ee7c8c3aa1e20937c22"}
Oct 29 22:14:36.266298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:36.266158    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7b6b64f55f-9l4xp" event={"ID":"19e92d72-5f01-4c0f-9739-e094b74d5825","Type":"ContainerStarted","Data":"abc5735b5fa7eb315dae76e7759be72eb4e632e536a05b213fcf968cff4eb52c"}
Oct 29 22:14:37.028038 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:37.028008    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:37.030281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:37.030258    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:37.269168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:37.269139    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:37.270217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:37.270197    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7b6b64f55f-9l4xp"
Oct 29 22:14:39.014512 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:39.014482    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:39.135293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:39.135261    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:39.268807 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:39.268682    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:40.251598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:40.251570    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5sn5b"
Oct 29 22:14:40.998164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:40.998136    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-99lmd"
Oct 29 22:14:41.952432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:41.952389    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:42.063879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:42.063840    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:44.200947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.200923    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:14:44.403668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.403618    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:44.531259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.531169    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:44.668932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.668892    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:44.683241 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.683212    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8stqn"]
Oct 29 22:14:44.718949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.718930    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.756521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.756476    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 22:14:44.756656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.756640    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 22:14:44.758920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.758904    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-wtmp\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.759010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.758933    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-accelerators-collector-config\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.759010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.758953    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-textfile\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.759010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.758971    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-tls\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.759105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.759073    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-root\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.759141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.759106    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.759141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.759135    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t7nv\" (UniqueName: \"kubernetes.io/projected/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-kube-api-access-9t7nv\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.759214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.759184    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-sys\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.759249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.759220    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-metrics-client-ca\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.759714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.759694    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mlg6x\""
Oct 29 22:14:44.759828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.759704    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 22:14:44.759828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.759742    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 22:14:44.795170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.795124    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:44.860300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860269    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-textfile\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.860300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860307    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-tls\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.860468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860341    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-root\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.860468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860357    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.860468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860387    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t7nv\" (UniqueName: \"kubernetes.io/projected/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-kube-api-access-9t7nv\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.860468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860402    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-sys\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.860468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860416    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-root\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.860731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860426    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-metrics-client-ca\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.860731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860563    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-wtmp\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.860731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860576    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-textfile\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.860731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860576    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-sys\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.860731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860601    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-accelerators-collector-config\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.860731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.860714    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-wtmp\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.861076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.861053    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-accelerators-collector-config\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.861134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.861087    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-metrics-client-ca\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.862675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.862655    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-tls\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.862757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.862673    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.880830 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.880812    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t7nv\" (UniqueName: \"kubernetes.io/projected/8fd7ac88-64eb-4ae3-a3bc-587cf3b365be-kube-api-access-9t7nv\") pod \"node-exporter-8stqn\" (UID: \"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be\") " pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:44.912482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:44.912456    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:45.030716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:45.030689    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:45.071802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:45.071751    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8stqn"
Oct 29 22:14:45.109458 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:14:45.109434    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd7ac88_64eb_4ae3_a3bc_587cf3b365be.slice/crio-b9aa99debe5b3ccb2ecd4b3f08f54f72385d8c71b9ccf5dde0212d3c1273fa94 WatchSource:0}: Error finding container b9aa99debe5b3ccb2ecd4b3f08f54f72385d8c71b9ccf5dde0212d3c1273fa94: Status 404 returned error can't find the container with id b9aa99debe5b3ccb2ecd4b3f08f54f72385d8c71b9ccf5dde0212d3c1273fa94
Oct 29 22:14:45.163168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:45.163141    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:45.283433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:45.283411    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:45.298455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:45.298430    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8stqn" event={"ID":"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be","Type":"ContainerStarted","Data":"b9aa99debe5b3ccb2ecd4b3f08f54f72385d8c71b9ccf5dde0212d3c1273fa94"}
Oct 29 22:14:45.653090 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:45.653064    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:14:45.761339 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:45.761299    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:45.899534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:45.899503    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:46.025912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:46.025879    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:47.307924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:47.307883    4860 generic.go:356] "Generic (PLEG): container finished" podID="8fd7ac88-64eb-4ae3-a3bc-587cf3b365be" containerID="02d799e367f34593b08172c066e5fadbacfb603297106dd2c7add0a5410287d4" exitCode=0
Oct 29 22:14:47.308271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:47.307970    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8stqn" event={"ID":"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be","Type":"ContainerDied","Data":"02d799e367f34593b08172c066e5fadbacfb603297106dd2c7add0a5410287d4"}
Oct 29 22:14:47.896465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:47.896434    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:48.003384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:48.003358    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:48.052850 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:14:48.052820    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:14:48.117198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:48.117169    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:48.313042 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:48.313008    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8stqn" event={"ID":"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be","Type":"ContainerStarted","Data":"3f011a24a70a75110978a89bc55777c663c4d0ed6d4326f3ff0baced44d85c1b"}
Oct 29 22:14:48.313555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:48.313057    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8stqn" event={"ID":"8fd7ac88-64eb-4ae3-a3bc-587cf3b365be","Type":"ContainerStarted","Data":"b28663851d7c31e941947e00025909064d46c91a46d2125278d42c6b066fb237"}
Oct 29 22:14:48.385117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:48.385076    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8stqn" podStartSLOduration=2.948376918 podStartE2EDuration="4.385063251s" totalImagesPullingTime="1.346543728s" totalInitContainerRuntime="90.142605ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:48.371357954 +0000 UTC m=+30.396134085" watchObservedRunningTime="2026-10-29 22:14:48.385063251 +0000 UTC m=+30.409839376"
Oct 29 22:14:49.493113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:49.493079    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:49.603434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:49.603404    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:49.724198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:49.724170    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:51.417777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:51.417745    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:51.580972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:51.580944    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sd9mh"
Oct 29 22:14:51.583354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:51.583301    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:51.724333 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:51.724281    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:52.716033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:52.716000    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:52.832575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:52.832547    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:56.604543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:56.604512    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:56.712691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:56.712651    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:56.951752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:56.951726    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:57.090505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:57.090479    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:58.433922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:58.433896    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f8746f6cf-j8g6f"]
Oct 29 22:14:58.559574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:58.559530    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:58.671480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:58.671451    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:14:58.875570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:14:58.875506    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:00.668582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.668501    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd" podUID="d0f94ef8-7c40-437b-a27e-36583be8bfba" containerName="registry" containerID="cri-o://9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c" gracePeriod=30
Oct 29 22:15:00.908432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.908411    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:15:00.984756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.984730    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-kube-api-access-tg246\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-kube-api-access-tg246\") pod \"d0f94ef8-7c40-437b-a27e-36583be8bfba\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") "
Oct 29 22:15:00.984954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.984774    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-installation-pull-secrets\") pod \"d0f94ef8-7c40-437b-a27e-36583be8bfba\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") "
Oct 29 22:15:00.984954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.984805    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f94ef8-7c40-437b-a27e-36583be8bfba-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0f94ef8-7c40-437b-a27e-36583be8bfba-ca-trust-extracted\") pod \"d0f94ef8-7c40-437b-a27e-36583be8bfba\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") "
Oct 29 22:15:00.984954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.984850    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-certificates\") pod \"d0f94ef8-7c40-437b-a27e-36583be8bfba\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") "
Oct 29 22:15:00.984954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.984877    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-bound-sa-token\") pod \"d0f94ef8-7c40-437b-a27e-36583be8bfba\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") "
Oct 29 22:15:00.984954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.984923    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-trusted-ca\") pod \"d0f94ef8-7c40-437b-a27e-36583be8bfba\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") "
Oct 29 22:15:00.984954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.984948    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") pod \"d0f94ef8-7c40-437b-a27e-36583be8bfba\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") "
Oct 29 22:15:00.985400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.985011    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-image-registry-private-configuration\") pod \"d0f94ef8-7c40-437b-a27e-36583be8bfba\" (UID: \"d0f94ef8-7c40-437b-a27e-36583be8bfba\") "
Oct 29 22:15:00.985461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.985319    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-certificates" pod "d0f94ef8-7c40-437b-a27e-36583be8bfba" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:00.985606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.985566    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-trusted-ca" pod "d0f94ef8-7c40-437b-a27e-36583be8bfba" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:00.988110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.988078    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-installation-pull-secrets" pod "d0f94ef8-7c40-437b-a27e-36583be8bfba" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:00.988497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.988467    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-image-registry-private-configuration" pod "d0f94ef8-7c40-437b-a27e-36583be8bfba" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:00.988497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.988483    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls" pod "d0f94ef8-7c40-437b-a27e-36583be8bfba" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:00.988764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.988717    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-bound-sa-token" pod "d0f94ef8-7c40-437b-a27e-36583be8bfba" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:00.988889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.988831    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-kube-api-access-tg246" pod "d0f94ef8-7c40-437b-a27e-36583be8bfba" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba"). InnerVolumeSpecName "kube-api-access-tg246". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:00.992251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:00.992199    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f94ef8-7c40-437b-a27e-36583be8bfba-ca-trust-extracted" pod "d0f94ef8-7c40-437b-a27e-36583be8bfba" (UID: "d0f94ef8-7c40-437b-a27e-36583be8bfba"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:01.086253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.086223    4860 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-trusted-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:01.086253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.086248    4860 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:01.086253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.086259    4860 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-image-registry-private-configuration\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:01.086478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.086270    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg246\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-kube-api-access-tg246\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:01.086478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.086280    4860 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0f94ef8-7c40-437b-a27e-36583be8bfba-installation-pull-secrets\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:01.086478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.086288    4860 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0f94ef8-7c40-437b-a27e-36583be8bfba-ca-trust-extracted\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:01.086478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.086297    4860 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0f94ef8-7c40-437b-a27e-36583be8bfba-registry-certificates\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:01.086478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.086306    4860 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0f94ef8-7c40-437b-a27e-36583be8bfba-bound-sa-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:01.360138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.360054    4860 generic.go:356] "Generic (PLEG): container finished" podID="d0f94ef8-7c40-437b-a27e-36583be8bfba" containerID="9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c" exitCode=0
Oct 29 22:15:01.360138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.360112    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd"
Oct 29 22:15:01.360308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.360113    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd" event={"ID":"d0f94ef8-7c40-437b-a27e-36583be8bfba","Type":"ContainerDied","Data":"9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c"}
Oct 29 22:15:01.360308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.360235    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9f64d5bbb-8pnfd" event={"ID":"d0f94ef8-7c40-437b-a27e-36583be8bfba","Type":"ContainerDied","Data":"d212502e014b15160dee880b88cb738266d4c04e2e4d3a9a06358edc53432175"}
Oct 29 22:15:01.360308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.360267    4860 scope.go:122] "RemoveContainer" containerID="9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c"
Oct 29 22:15:01.372138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.372120    4860 scope.go:122] "RemoveContainer" containerID="9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c"
Oct 29 22:15:01.372450 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:15:01.372423    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c\": container with ID starting with 9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c not found: ID does not exist" containerID="9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c"
Oct 29 22:15:01.372553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.372458    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c"} err="failed to get container status \"9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c\": rpc error: code = NotFound desc = could not find container \"9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c\": container with ID starting with 9010551ffd1eea11f7d9a068f313a41d72e40caf1b6ce8f1b97e4f2192dff88c not found: ID does not exist"
Oct 29 22:15:01.414679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.414657    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9f64d5bbb-8pnfd"]
Oct 29 22:15:01.428598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:01.428573    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9f64d5bbb-8pnfd"]
Oct 29 22:15:02.109995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:02.109959    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f94ef8-7c40-437b-a27e-36583be8bfba" path="/var/lib/kubelet/pods/d0f94ef8-7c40-437b-a27e-36583be8bfba/volumes"
Oct 29 22:15:03.081745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:03.081715    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:03.200779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:03.200749    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:06.618511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:06.618478    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:06.737854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:06.737820    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:07.181085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:07.181057    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:07.303403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:07.303377    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:08.415666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:08.415638    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:08.523111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:08.523083    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:08.672711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:08.672640    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:08.809191 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:08.809164    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:11.813188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:11.813157    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:11.923798 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:11.923769    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:12.028421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:12.028395    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:18.053255 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:15:18.053220    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:15:23.456382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.456311    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f" podUID="0da83403-18e0-4abd-8bbb-9323e80a3959" containerName="registry" containerID="cri-o://208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea" gracePeriod=30
Oct 29 22:15:23.693480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.693459    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:15:23.855017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.854959    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-image-registry-private-configuration\") pod \"0da83403-18e0-4abd-8bbb-9323e80a3959\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") "
Oct 29 22:15:23.855134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.855026    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-trusted-ca\") pod \"0da83403-18e0-4abd-8bbb-9323e80a3959\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") "
Oct 29 22:15:23.855134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.855065    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-installation-pull-secrets\") pod \"0da83403-18e0-4abd-8bbb-9323e80a3959\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") "
Oct 29 22:15:23.855134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.855108    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") pod \"0da83403-18e0-4abd-8bbb-9323e80a3959\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") "
Oct 29 22:15:23.855253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.855147    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-certificates\") pod \"0da83403-18e0-4abd-8bbb-9323e80a3959\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") "
Oct 29 22:15:23.855253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.855191    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-bound-sa-token\") pod \"0da83403-18e0-4abd-8bbb-9323e80a3959\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") "
Oct 29 22:15:23.855253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.855223    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-kube-api-access-gwxqn\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-kube-api-access-gwxqn\") pod \"0da83403-18e0-4abd-8bbb-9323e80a3959\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") "
Oct 29 22:15:23.855425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.855299    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da83403-18e0-4abd-8bbb-9323e80a3959-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0da83403-18e0-4abd-8bbb-9323e80a3959-ca-trust-extracted\") pod \"0da83403-18e0-4abd-8bbb-9323e80a3959\" (UID: \"0da83403-18e0-4abd-8bbb-9323e80a3959\") "
Oct 29 22:15:23.855558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.855529    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-trusted-ca" pod "0da83403-18e0-4abd-8bbb-9323e80a3959" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:23.855695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.855672    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-certificates" pod "0da83403-18e0-4abd-8bbb-9323e80a3959" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:23.857723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.857694    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls" pod "0da83403-18e0-4abd-8bbb-9323e80a3959" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:23.857828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.857731    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-kube-api-access-gwxqn" pod "0da83403-18e0-4abd-8bbb-9323e80a3959" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959"). InnerVolumeSpecName "kube-api-access-gwxqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:23.857828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.857812    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-image-registry-private-configuration" pod "0da83403-18e0-4abd-8bbb-9323e80a3959" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:23.858062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.858040    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-bound-sa-token" pod "0da83403-18e0-4abd-8bbb-9323e80a3959" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:23.858150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.858068    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-installation-pull-secrets" pod "0da83403-18e0-4abd-8bbb-9323e80a3959" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:23.862335 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.862306    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da83403-18e0-4abd-8bbb-9323e80a3959-ca-trust-extracted" pod "0da83403-18e0-4abd-8bbb-9323e80a3959" (UID: "0da83403-18e0-4abd-8bbb-9323e80a3959"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:23.956458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.956436    4860 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:23.956458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.956456    4860 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-registry-certificates\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:23.956580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.956466    4860 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-bound-sa-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:23.956580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.956477    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwxqn\" (UniqueName: \"kubernetes.io/projected/0da83403-18e0-4abd-8bbb-9323e80a3959-kube-api-access-gwxqn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:23.956580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.956486    4860 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0da83403-18e0-4abd-8bbb-9323e80a3959-ca-trust-extracted\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:23.956580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.956496    4860 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-image-registry-private-configuration\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:23.956580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.956505    4860 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0da83403-18e0-4abd-8bbb-9323e80a3959-trusted-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:23.956580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:23.956514    4860 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0da83403-18e0-4abd-8bbb-9323e80a3959-installation-pull-secrets\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:15:24.445610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:24.445582    4860 generic.go:356] "Generic (PLEG): container finished" podID="0da83403-18e0-4abd-8bbb-9323e80a3959" containerID="208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea" exitCode=0
Oct 29 22:15:24.445758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:24.445627    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f"
Oct 29 22:15:24.445758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:24.445650    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f" event={"ID":"0da83403-18e0-4abd-8bbb-9323e80a3959","Type":"ContainerDied","Data":"208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea"}
Oct 29 22:15:24.445758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:24.445690    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8746f6cf-j8g6f" event={"ID":"0da83403-18e0-4abd-8bbb-9323e80a3959","Type":"ContainerDied","Data":"584141ef3b8938ec229fa962a0d394109dbcbc8b1896aadc081a03f92df85cbc"}
Oct 29 22:15:24.445758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:24.445709    4860 scope.go:122] "RemoveContainer" containerID="208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea"
Oct 29 22:15:24.453296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:24.453280    4860 scope.go:122] "RemoveContainer" containerID="208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea"
Oct 29 22:15:24.453587 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:15:24.453566    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea\": container with ID starting with 208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea not found: ID does not exist" containerID="208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea"
Oct 29 22:15:24.453650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:24.453593    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea"} err="failed to get container status \"208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea\": rpc error: code = NotFound desc = could not find container \"208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea\": container with ID starting with 208a62d2f860f2138ffcae7685f8d5df5798ce185f3cc164f9bed9ad0510d0ea not found: ID does not exist"
Oct 29 22:15:24.513890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:24.513864    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f8746f6cf-j8g6f"]
Oct 29 22:15:24.527367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:24.527346    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f8746f6cf-j8g6f"]
Oct 29 22:15:26.108782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:26.108749    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da83403-18e0-4abd-8bbb-9323e80a3959" path="/var/lib/kubelet/pods/0da83403-18e0-4abd-8bbb-9323e80a3959/volumes"
Oct 29 22:15:26.941003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:26.940976    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:27.051134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:27.051104    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:30.419826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:30.419791    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:30.528597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:30.528566    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:31.039470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:31.039432    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:31.150707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:31.150676    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:32.694615 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:32.694586    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:32.805800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:32.805772    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:34.623234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:34.623206    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:43.052012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:43.051974    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:43.159375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:43.159315    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:44.512664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:44.512633    4860 generic.go:356] "Generic (PLEG): container finished" podID="9f465b95-5af5-4d92-921b-24b0800934a0" containerID="c715ed8c9ef4252f38eebce796362d32340fb1610083d5c998624563f8507041" exitCode=0
Oct 29 22:15:44.513141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:44.512711    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg" event={"ID":"9f465b95-5af5-4d92-921b-24b0800934a0","Type":"ContainerDied","Data":"c715ed8c9ef4252f38eebce796362d32340fb1610083d5c998624563f8507041"}
Oct 29 22:15:44.513141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:44.513058    4860 scope.go:122] "RemoveContainer" containerID="c715ed8c9ef4252f38eebce796362d32340fb1610083d5c998624563f8507041"
Oct 29 22:15:45.095921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:45.095888    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:45.209622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:45.209586    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:45.517737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:45.517705    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jmtsg" event={"ID":"9f465b95-5af5-4d92-921b-24b0800934a0","Type":"ContainerStarted","Data":"9c3659e60586311da2541629128c7de835470dd37c98d124be33c814af35e9fd"}
Oct 29 22:15:48.052760 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:15:48.052726    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:15:49.533493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:49.533439    4860 generic.go:356] "Generic (PLEG): container finished" podID="fb84513d-ca51-4920-bfc5-4240aa395064" containerID="2f61f374adcab3f31219b2e24b52ad05ace83d28684e07e6d9dba26cdfff28d0" exitCode=0
Oct 29 22:15:49.533899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:49.533501    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wfpts" event={"ID":"fb84513d-ca51-4920-bfc5-4240aa395064","Type":"ContainerDied","Data":"2f61f374adcab3f31219b2e24b52ad05ace83d28684e07e6d9dba26cdfff28d0"}
Oct 29 22:15:49.533899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:49.533773    4860 scope.go:122] "RemoveContainer" containerID="2f61f374adcab3f31219b2e24b52ad05ace83d28684e07e6d9dba26cdfff28d0"
Oct 29 22:15:50.340210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.340176    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-nkcfh-l6q7h"]
Oct 29 22:15:50.340497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.340475    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0da83403-18e0-4abd-8bbb-9323e80a3959" containerName="registry"
Oct 29 22:15:50.340497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.340494    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da83403-18e0-4abd-8bbb-9323e80a3959" containerName="registry"
Oct 29 22:15:50.340674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.340512    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d0f94ef8-7c40-437b-a27e-36583be8bfba" containerName="registry"
Oct 29 22:15:50.340674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.340519    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f94ef8-7c40-437b-a27e-36583be8bfba" containerName="registry"
Oct 29 22:15:50.340674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.340612    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da83403-18e0-4abd-8bbb-9323e80a3959" containerName="registry"
Oct 29 22:15:50.340674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.340625    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f94ef8-7c40-437b-a27e-36583be8bfba" containerName="registry"
Oct 29 22:15:50.343566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.343546    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.360409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.360387    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nkcfh-l6q7h"]
Oct 29 22:15:50.534446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.534408    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1ba99894-60f7-4633-948d-e3869f51fab8-archives-path\") pod \"periodic-gathering-nkcfh-l6q7h\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") " pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.534862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.534494    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-trusted-ca-bundle\") pod \"periodic-gathering-nkcfh-l6q7h\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") " pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.534862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.534550    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-service-ca-bundle\") pod \"periodic-gathering-nkcfh-l6q7h\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") " pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.534862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.534680    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x8zv\" (UniqueName: \"kubernetes.io/projected/1ba99894-60f7-4633-948d-e3869f51fab8-kube-api-access-4x8zv\") pod \"periodic-gathering-nkcfh-l6q7h\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") " pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.538981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.538956    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wfpts" event={"ID":"fb84513d-ca51-4920-bfc5-4240aa395064","Type":"ContainerStarted","Data":"34668df70e3a501b04cc40c9af71757d0ada611c03138dbb174f8425e1dbcdc4"}
Oct 29 22:15:50.635208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.635136    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-service-ca-bundle\") pod \"periodic-gathering-nkcfh-l6q7h\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") " pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.635208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.635181    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8zv\" (UniqueName: \"kubernetes.io/projected/1ba99894-60f7-4633-948d-e3869f51fab8-kube-api-access-4x8zv\") pod \"periodic-gathering-nkcfh-l6q7h\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") " pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.635458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.635230    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1ba99894-60f7-4633-948d-e3869f51fab8-archives-path\") pod \"periodic-gathering-nkcfh-l6q7h\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") " pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.635458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.635249    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-trusted-ca-bundle\") pod \"periodic-gathering-nkcfh-l6q7h\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") " pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.635636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.635612    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1ba99894-60f7-4633-948d-e3869f51fab8-archives-path\") pod \"periodic-gathering-nkcfh-l6q7h\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") " pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.635887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.635864    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-service-ca-bundle\") pod \"periodic-gathering-nkcfh-l6q7h\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") " pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.636337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.636304    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-trusted-ca-bundle\") pod \"periodic-gathering-nkcfh-l6q7h\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") " pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.656650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.656628    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8zv\" (UniqueName: \"kubernetes.io/projected/1ba99894-60f7-4633-948d-e3869f51fab8-kube-api-access-4x8zv\") pod \"periodic-gathering-nkcfh-l6q7h\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") " pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:50.954766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:50.954741    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:15:51.241887 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:15:51.241857    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba99894_60f7_4633_948d_e3869f51fab8.slice/crio-4b4ad98d4dc69a196db0adfbb829b2d56f14c946a25903aa4a7bbed5cc8d7c6e WatchSource:0}: Error finding container 4b4ad98d4dc69a196db0adfbb829b2d56f14c946a25903aa4a7bbed5cc8d7c6e: Status 404 returned error can't find the container with id 4b4ad98d4dc69a196db0adfbb829b2d56f14c946a25903aa4a7bbed5cc8d7c6e
Oct 29 22:15:51.242084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:51.242065    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nkcfh-l6q7h"]
Oct 29 22:15:51.543448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:51.543367    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nkcfh-l6q7h" event={"ID":"1ba99894-60f7-4633-948d-e3869f51fab8","Type":"ContainerStarted","Data":"0f1a93f1ccf7eb650f63b3dfb43b9fc4c304c55048640dc418b5264a1686bd97"}
Oct 29 22:15:51.543448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:51.543422    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nkcfh-l6q7h" event={"ID":"1ba99894-60f7-4633-948d-e3869f51fab8","Type":"ContainerStarted","Data":"4b4ad98d4dc69a196db0adfbb829b2d56f14c946a25903aa4a7bbed5cc8d7c6e"}
Oct 29 22:15:51.582595 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:51.582547    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-nkcfh-l6q7h" podStartSLOduration=1.582534629 podStartE2EDuration="1.582534629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:51.582160439 +0000 UTC m=+93.606936582" watchObservedRunningTime="2026-10-29 22:15:51.582534629 +0000 UTC m=+93.607310741"
Oct 29 22:15:54.556912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:54.556877    4860 generic.go:356] "Generic (PLEG): container finished" podID="a342967b-704c-4947-92fa-e2dc475675a0" containerID="4cb9f8be95497267181303b5dd9c87256d46723aac4132c8d7c9cfb4c1ff5bee" exitCode=0
Oct 29 22:15:54.557284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:54.556950    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6" event={"ID":"a342967b-704c-4947-92fa-e2dc475675a0","Type":"ContainerDied","Data":"4cb9f8be95497267181303b5dd9c87256d46723aac4132c8d7c9cfb4c1ff5bee"}
Oct 29 22:15:54.557284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:54.557265    4860 scope.go:122] "RemoveContainer" containerID="4cb9f8be95497267181303b5dd9c87256d46723aac4132c8d7c9cfb4c1ff5bee"
Oct 29 22:15:55.562540 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:55.562506    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-s2cf6" event={"ID":"a342967b-704c-4947-92fa-e2dc475675a0","Type":"ContainerStarted","Data":"4622bf8ba4074b1af4ee4c8e304b41d2c24ab659afd6684126992e3abfaf025f"}
Oct 29 22:15:56.945734 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:56.945704    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:15:57.061068 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:15:57.061037    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:08.817440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:16:08.817397    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:16:18.053317 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:16:18.053274    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:16:48.053939 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:16:48.053863    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:17:08.902526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:08.902489    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:17:14.834189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:14.834109    4860 generic.go:356] "Generic (PLEG): container finished" podID="1ba99894-60f7-4633-948d-e3869f51fab8" containerID="0f1a93f1ccf7eb650f63b3dfb43b9fc4c304c55048640dc418b5264a1686bd97" exitCode=0
Oct 29 22:17:14.834603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:14.834181    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nkcfh-l6q7h" event={"ID":"1ba99894-60f7-4633-948d-e3869f51fab8","Type":"ContainerDied","Data":"0f1a93f1ccf7eb650f63b3dfb43b9fc4c304c55048640dc418b5264a1686bd97"}
Oct 29 22:17:15.963142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:15.963117    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:17:16.132946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.132877    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba99894-60f7-4633-948d-e3869f51fab8-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1ba99894-60f7-4633-948d-e3869f51fab8-archives-path\") pod \"1ba99894-60f7-4633-948d-e3869f51fab8\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") "
Oct 29 22:17:16.132946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.132922    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba99894-60f7-4633-948d-e3869f51fab8-kube-api-access-4x8zv\" (UniqueName: \"kubernetes.io/projected/1ba99894-60f7-4633-948d-e3869f51fab8-kube-api-access-4x8zv\") pod \"1ba99894-60f7-4633-948d-e3869f51fab8\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") "
Oct 29 22:17:16.133128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.132951    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-service-ca-bundle\") pod \"1ba99894-60f7-4633-948d-e3869f51fab8\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") "
Oct 29 22:17:16.133128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.132993    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-trusted-ca-bundle\") pod \"1ba99894-60f7-4633-948d-e3869f51fab8\" (UID: \"1ba99894-60f7-4633-948d-e3869f51fab8\") "
Oct 29 22:17:16.133272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.133245    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba99894-60f7-4633-948d-e3869f51fab8-archives-path" pod "1ba99894-60f7-4633-948d-e3869f51fab8" (UID: "1ba99894-60f7-4633-948d-e3869f51fab8"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:17:16.133355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.133289    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-service-ca-bundle" pod "1ba99894-60f7-4633-948d-e3869f51fab8" (UID: "1ba99894-60f7-4633-948d-e3869f51fab8"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:17:16.133416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.133367    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-trusted-ca-bundle" pod "1ba99894-60f7-4633-948d-e3869f51fab8" (UID: "1ba99894-60f7-4633-948d-e3869f51fab8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:17:16.135103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.135085    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba99894-60f7-4633-948d-e3869f51fab8-kube-api-access-4x8zv" pod "1ba99894-60f7-4633-948d-e3869f51fab8" (UID: "1ba99894-60f7-4633-948d-e3869f51fab8"). InnerVolumeSpecName "kube-api-access-4x8zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:17:16.233756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.233715    4860 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-service-ca-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:17:16.233756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.233748    4860 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ba99894-60f7-4633-948d-e3869f51fab8-trusted-ca-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:17:16.233756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.233757    4860 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1ba99894-60f7-4633-948d-e3869f51fab8-archives-path\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:17:16.233756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.233766    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x8zv\" (UniqueName: \"kubernetes.io/projected/1ba99894-60f7-4633-948d-e3869f51fab8-kube-api-access-4x8zv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:17:16.842159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.842127    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nkcfh-l6q7h" event={"ID":"1ba99894-60f7-4633-948d-e3869f51fab8","Type":"ContainerDied","Data":"4b4ad98d4dc69a196db0adfbb829b2d56f14c946a25903aa4a7bbed5cc8d7c6e"}
Oct 29 22:17:16.842159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.842158    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nkcfh-l6q7h"
Oct 29 22:17:16.842367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:16.842172    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b4ad98d4dc69a196db0adfbb829b2d56f14c946a25903aa4a7bbed5cc8d7c6e"
Oct 29 22:17:18.053369 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:17:18.053317    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:17:21.932780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:21.932730    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:17:44.120981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.120948    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:17:44.267871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.267838    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:17:44.723068 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.723036    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"]
Oct 29 22:17:44.723318 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.723306    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1ba99894-60f7-4633-948d-e3869f51fab8" containerName="insights-gathering"
Oct 29 22:17:44.723318 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.723319    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba99894-60f7-4633-948d-e3869f51fab8" containerName="insights-gathering"
Oct 29 22:17:44.723418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.723394    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba99894-60f7-4633-948d-e3869f51fab8" containerName="insights-gathering"
Oct 29 22:17:44.726444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.726422    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:17:44.758710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.758679    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 29 22:17:44.758827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.758756    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-vflnl\""
Oct 29 22:17:44.758827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.758769    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 29 22:17:44.759508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.759490    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"]
Oct 29 22:17:44.811979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.811955    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrvgm\" (UniqueName: \"kubernetes.io/projected/0dae9652-9144-4915-8fd9-64ddae7674ee-kube-api-access-rrvgm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5\" (UID: \"0dae9652-9144-4915-8fd9-64ddae7674ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:17:44.812079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.811983    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5\" (UID: \"0dae9652-9144-4915-8fd9-64ddae7674ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:17:44.812079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.812001    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5\" (UID: \"0dae9652-9144-4915-8fd9-64ddae7674ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:17:44.912796 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.912740    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrvgm\" (UniqueName: \"kubernetes.io/projected/0dae9652-9144-4915-8fd9-64ddae7674ee-kube-api-access-rrvgm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5\" (UID: \"0dae9652-9144-4915-8fd9-64ddae7674ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:17:44.912796 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.912797    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5\" (UID: \"0dae9652-9144-4915-8fd9-64ddae7674ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:17:44.913034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.912825    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5\" (UID: \"0dae9652-9144-4915-8fd9-64ddae7674ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:17:44.913279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.913258    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5\" (UID: \"0dae9652-9144-4915-8fd9-64ddae7674ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:17:44.913386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.913356    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5\" (UID: \"0dae9652-9144-4915-8fd9-64ddae7674ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:17:44.933449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:44.933430    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrvgm\" (UniqueName: \"kubernetes.io/projected/0dae9652-9144-4915-8fd9-64ddae7674ee-kube-api-access-rrvgm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5\" (UID: \"0dae9652-9144-4915-8fd9-64ddae7674ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:17:45.041422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:45.041369    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:17:45.332534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:45.332510    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"]
Oct 29 22:17:45.334537 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:17:45.334501    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dae9652_9144_4915_8fd9_64ddae7674ee.slice/crio-c2c9163b5161612f413b944520344a9d03e90dc7c4585a14022309a66576d83b WatchSource:0}: Error finding container c2c9163b5161612f413b944520344a9d03e90dc7c4585a14022309a66576d83b: Status 404 returned error can't find the container with id c2c9163b5161612f413b944520344a9d03e90dc7c4585a14022309a66576d83b
Oct 29 22:17:45.952580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:45.952540    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5" event={"ID":"0dae9652-9144-4915-8fd9-64ddae7674ee","Type":"ContainerStarted","Data":"c2c9163b5161612f413b944520344a9d03e90dc7c4585a14022309a66576d83b"}
Oct 29 22:17:48.060305 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:17:48.060261    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:17:50.393917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:50.393887    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:17:50.506919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:50.506876    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:17:50.973010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:50.972975    4860 generic.go:356] "Generic (PLEG): container finished" podID="0dae9652-9144-4915-8fd9-64ddae7674ee" containerID="6c9369b8d61e7869f176bb5c5aabb6af177b6180969adc86321ad75b8efd24fa" exitCode=0
Oct 29 22:17:50.973172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:50.973061    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5" event={"ID":"0dae9652-9144-4915-8fd9-64ddae7674ee","Type":"ContainerDied","Data":"6c9369b8d61e7869f176bb5c5aabb6af177b6180969adc86321ad75b8efd24fa"}
Oct 29 22:17:53.987279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:53.987237    4860 generic.go:356] "Generic (PLEG): container finished" podID="0dae9652-9144-4915-8fd9-64ddae7674ee" containerID="a17e3fe5e6fe4b3b4f979dae45b9acef284b2cf478e858a88794a92ef07c12b2" exitCode=0
Oct 29 22:17:53.987673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:17:53.987294    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5" event={"ID":"0dae9652-9144-4915-8fd9-64ddae7674ee","Type":"ContainerDied","Data":"a17e3fe5e6fe4b3b4f979dae45b9acef284b2cf478e858a88794a92ef07c12b2"}
Oct 29 22:18:00.012080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:00.012043    4860 generic.go:356] "Generic (PLEG): container finished" podID="0dae9652-9144-4915-8fd9-64ddae7674ee" containerID="05c599548255a78f8d2e8917d0f9836c7ebf1556ebb17d33f1d451577a80ccb8" exitCode=0
Oct 29 22:18:00.012523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:00.012126    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5" event={"ID":"0dae9652-9144-4915-8fd9-64ddae7674ee","Type":"ContainerDied","Data":"05c599548255a78f8d2e8917d0f9836c7ebf1556ebb17d33f1d451577a80ccb8"}
Oct 29 22:18:01.134810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:01.134791    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:18:01.238197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:01.238174    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dae9652-9144-4915-8fd9-64ddae7674ee-kube-api-access-rrvgm\" (UniqueName: \"kubernetes.io/projected/0dae9652-9144-4915-8fd9-64ddae7674ee-kube-api-access-rrvgm\") pod \"0dae9652-9144-4915-8fd9-64ddae7674ee\" (UID: \"0dae9652-9144-4915-8fd9-64ddae7674ee\") "
Oct 29 22:18:01.238302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:01.238231    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-bundle\") pod \"0dae9652-9144-4915-8fd9-64ddae7674ee\" (UID: \"0dae9652-9144-4915-8fd9-64ddae7674ee\") "
Oct 29 22:18:01.238390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:01.238306    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-util\" (UniqueName: \"kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-util\") pod \"0dae9652-9144-4915-8fd9-64ddae7674ee\" (UID: \"0dae9652-9144-4915-8fd9-64ddae7674ee\") "
Oct 29 22:18:01.238859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:01.238836    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-bundle" pod "0dae9652-9144-4915-8fd9-64ddae7674ee" (UID: "0dae9652-9144-4915-8fd9-64ddae7674ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:18:01.240296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:01.240266    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dae9652-9144-4915-8fd9-64ddae7674ee-kube-api-access-rrvgm" pod "0dae9652-9144-4915-8fd9-64ddae7674ee" (UID: "0dae9652-9144-4915-8fd9-64ddae7674ee"). InnerVolumeSpecName "kube-api-access-rrvgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:18:01.243278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:01.243255    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-util" pod "0dae9652-9144-4915-8fd9-64ddae7674ee" (UID: "0dae9652-9144-4915-8fd9-64ddae7674ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:18:01.339705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:01.339657    4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:18:01.339705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:01.339679    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrvgm\" (UniqueName: \"kubernetes.io/projected/0dae9652-9144-4915-8fd9-64ddae7674ee-kube-api-access-rrvgm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:18:01.339705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:01.339688    4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dae9652-9144-4915-8fd9-64ddae7674ee-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:18:02.021421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:02.021387    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5" event={"ID":"0dae9652-9144-4915-8fd9-64ddae7674ee","Type":"ContainerDied","Data":"c2c9163b5161612f413b944520344a9d03e90dc7c4585a14022309a66576d83b"}
Oct 29 22:18:02.021586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:02.021435    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c9163b5161612f413b944520344a9d03e90dc7c4585a14022309a66576d83b"
Oct 29 22:18:02.021586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:02.021437    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5gws5"
Oct 29 22:18:05.713810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:05.713778    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:06.519904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:06.519872    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:07.572398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.572362    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9"]
Oct 29 22:18:07.572756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.572651    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0dae9652-9144-4915-8fd9-64ddae7674ee" containerName="util"
Oct 29 22:18:07.572756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.572662    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dae9652-9144-4915-8fd9-64ddae7674ee" containerName="util"
Oct 29 22:18:07.572756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.572682    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0dae9652-9144-4915-8fd9-64ddae7674ee" containerName="pull"
Oct 29 22:18:07.572756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.572687    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dae9652-9144-4915-8fd9-64ddae7674ee" containerName="pull"
Oct 29 22:18:07.572756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.572696    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0dae9652-9144-4915-8fd9-64ddae7674ee" containerName="extract"
Oct 29 22:18:07.572756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.572700    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dae9652-9144-4915-8fd9-64ddae7674ee" containerName="extract"
Oct 29 22:18:07.572756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.572747    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dae9652-9144-4915-8fd9-64ddae7674ee" containerName="extract"
Oct 29 22:18:07.606293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.606269    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9"]
Oct 29 22:18:07.606435 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.606376    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9"
Oct 29 22:18:07.622768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.622743    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:18:07.622909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.622848    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 29 22:18:07.622909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.622874    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8drfc\""
Oct 29 22:18:07.684676 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.684652    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk56m\" (UniqueName: \"kubernetes.io/projected/547a0c90-9b24-4fb4-98aa-afe6936f0020-kube-api-access-sk56m\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-jh8p9\" (UID: \"547a0c90-9b24-4fb4-98aa-afe6936f0020\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9"
Oct 29 22:18:07.684765 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.684698    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/547a0c90-9b24-4fb4-98aa-afe6936f0020-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-jh8p9\" (UID: \"547a0c90-9b24-4fb4-98aa-afe6936f0020\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9"
Oct 29 22:18:07.785527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.785503    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/547a0c90-9b24-4fb4-98aa-afe6936f0020-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-jh8p9\" (UID: \"547a0c90-9b24-4fb4-98aa-afe6936f0020\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9"
Oct 29 22:18:07.785607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.785559    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk56m\" (UniqueName: \"kubernetes.io/projected/547a0c90-9b24-4fb4-98aa-afe6936f0020-kube-api-access-sk56m\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-jh8p9\" (UID: \"547a0c90-9b24-4fb4-98aa-afe6936f0020\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9"
Oct 29 22:18:07.785842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.785824    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/547a0c90-9b24-4fb4-98aa-afe6936f0020-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-jh8p9\" (UID: \"547a0c90-9b24-4fb4-98aa-afe6936f0020\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9"
Oct 29 22:18:07.805975 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.805953    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk56m\" (UniqueName: \"kubernetes.io/projected/547a0c90-9b24-4fb4-98aa-afe6936f0020-kube-api-access-sk56m\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-jh8p9\" (UID: \"547a0c90-9b24-4fb4-98aa-afe6936f0020\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9"
Oct 29 22:18:07.928631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:07.928582    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9"
Oct 29 22:18:08.264604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:08.264577    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9"]
Oct 29 22:18:08.265304 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:18:08.265281    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547a0c90_9b24_4fb4_98aa_afe6936f0020.slice/crio-c335829be0d632fb7e49ae7a9b05b92307ea207ba0bd9ebf187fb9d462131a41 WatchSource:0}: Error finding container c335829be0d632fb7e49ae7a9b05b92307ea207ba0bd9ebf187fb9d462131a41: Status 404 returned error can't find the container with id c335829be0d632fb7e49ae7a9b05b92307ea207ba0bd9ebf187fb9d462131a41
Oct 29 22:18:08.932455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:08.932422    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:09.043845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:09.043813    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:09.047830 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:09.047796    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9" event={"ID":"547a0c90-9b24-4fb4-98aa-afe6936f0020","Type":"ContainerStarted","Data":"c335829be0d632fb7e49ae7a9b05b92307ea207ba0bd9ebf187fb9d462131a41"}
Oct 29 22:18:09.719931 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:09.719879    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:09.833449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:09.833416    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:11.056962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.056926    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9" event={"ID":"547a0c90-9b24-4fb4-98aa-afe6936f0020","Type":"ContainerStarted","Data":"037258e9640759818d91d787fd05fe5212115836edc3e9a2115520286d542ec7"}
Oct 29 22:18:11.086576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.086540    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:11.130367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.130299    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jh8p9" podStartSLOduration=2.37643798 podStartE2EDuration="4.130287246s" totalImagesPullingTime="1.753849266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:11.116461026 +0000 UTC m=+233.141237159" watchObservedRunningTime="2026-10-29 22:18:11.130287246 +0000 UTC m=+233.155063377"
Oct 29 22:18:11.206358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.206312    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:11.255053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.255024    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"]
Oct 29 22:18:11.273719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.273700    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"
Oct 29 22:18:11.278454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.278431    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"]
Oct 29 22:18:11.289853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.289830    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qd4lp\""
Oct 29 22:18:11.289853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.289839    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 29 22:18:11.289997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.289867    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 29 22:18:11.309227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.309175    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e42bac5-f252-4c5f-9739-f0b300489d73-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-l6lbv\" (UID: \"0e42bac5-f252-4c5f-9739-f0b300489d73\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"
Oct 29 22:18:11.309227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.309217    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcc6v\" (UniqueName: \"kubernetes.io/projected/0e42bac5-f252-4c5f-9739-f0b300489d73-kube-api-access-mcc6v\") pod \"cert-manager-webhook-5dbbbf75cc-l6lbv\" (UID: \"0e42bac5-f252-4c5f-9739-f0b300489d73\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"
Oct 29 22:18:11.409512 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.409486    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e42bac5-f252-4c5f-9739-f0b300489d73-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-l6lbv\" (UID: \"0e42bac5-f252-4c5f-9739-f0b300489d73\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"
Oct 29 22:18:11.409611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.409525    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcc6v\" (UniqueName: \"kubernetes.io/projected/0e42bac5-f252-4c5f-9739-f0b300489d73-kube-api-access-mcc6v\") pod \"cert-manager-webhook-5dbbbf75cc-l6lbv\" (UID: \"0e42bac5-f252-4c5f-9739-f0b300489d73\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"
Oct 29 22:18:11.430919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.430880    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e42bac5-f252-4c5f-9739-f0b300489d73-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-l6lbv\" (UID: \"0e42bac5-f252-4c5f-9739-f0b300489d73\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"
Oct 29 22:18:11.431164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.431149    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcc6v\" (UniqueName: \"kubernetes.io/projected/0e42bac5-f252-4c5f-9739-f0b300489d73-kube-api-access-mcc6v\") pod \"cert-manager-webhook-5dbbbf75cc-l6lbv\" (UID: \"0e42bac5-f252-4c5f-9739-f0b300489d73\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"
Oct 29 22:18:11.518076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.518053    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:11.607958 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.607903    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"
Oct 29 22:18:11.690913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.690808    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:11.835221 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.835194    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:11.921385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:11.921352    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"]
Oct 29 22:18:11.921622 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:18:11.921601    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e42bac5_f252_4c5f_9739_f0b300489d73.slice/crio-563227a92177c4fd5e5f9bbe378ac91c731d25e7d93c54042edbe7d7a1aa2e80 WatchSource:0}: Error finding container 563227a92177c4fd5e5f9bbe378ac91c731d25e7d93c54042edbe7d7a1aa2e80: Status 404 returned error can't find the container with id 563227a92177c4fd5e5f9bbe378ac91c731d25e7d93c54042edbe7d7a1aa2e80
Oct 29 22:18:12.061483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.061454    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv" event={"ID":"0e42bac5-f252-4c5f-9739-f0b300489d73","Type":"ContainerStarted","Data":"563227a92177c4fd5e5f9bbe378ac91c731d25e7d93c54042edbe7d7a1aa2e80"}
Oct 29 22:18:12.089239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.089206    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:12.309116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.309086    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"]
Oct 29 22:18:12.343103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.343076    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"]
Oct 29 22:18:12.343235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.343173    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:12.374797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.374778    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 29 22:18:12.381782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.381764    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 29 22:18:12.381861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.381806    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-x54vw\""
Oct 29 22:18:12.417054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.417027    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg\" (UID: \"f0ad8058-a52d-4663-ba9a-389e0b237d29\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:12.417160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.417080    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxdzn\" (UniqueName: \"kubernetes.io/projected/f0ad8058-a52d-4663-ba9a-389e0b237d29-kube-api-access-pxdzn\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg\" (UID: \"f0ad8058-a52d-4663-ba9a-389e0b237d29\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:12.417232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.417222    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg\" (UID: \"f0ad8058-a52d-4663-ba9a-389e0b237d29\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:12.517969 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.517938    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg\" (UID: \"f0ad8058-a52d-4663-ba9a-389e0b237d29\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:12.518124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.517992    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg\" (UID: \"f0ad8058-a52d-4663-ba9a-389e0b237d29\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:12.518124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.518035    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxdzn\" (UniqueName: \"kubernetes.io/projected/f0ad8058-a52d-4663-ba9a-389e0b237d29-kube-api-access-pxdzn\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg\" (UID: \"f0ad8058-a52d-4663-ba9a-389e0b237d29\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:12.518420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.518392    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg\" (UID: \"f0ad8058-a52d-4663-ba9a-389e0b237d29\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:12.518420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.518406    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg\" (UID: \"f0ad8058-a52d-4663-ba9a-389e0b237d29\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:12.539189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.539164    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxdzn\" (UniqueName: \"kubernetes.io/projected/f0ad8058-a52d-4663-ba9a-389e0b237d29-kube-api-access-pxdzn\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg\" (UID: \"f0ad8058-a52d-4663-ba9a-389e0b237d29\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:12.688151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:12.688117    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:13.000578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:13.000543    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"]
Oct 29 22:18:13.001982 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:18:13.001943    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ad8058_a52d_4663_ba9a_389e0b237d29.slice/crio-21f26df1940c27eaa29319d39c391798b35ed979f587394aeba8f839d2edb8a1 WatchSource:0}: Error finding container 21f26df1940c27eaa29319d39c391798b35ed979f587394aeba8f839d2edb8a1: Status 404 returned error can't find the container with id 21f26df1940c27eaa29319d39c391798b35ed979f587394aeba8f839d2edb8a1
Oct 29 22:18:13.067857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:13.067831    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg" event={"ID":"f0ad8058-a52d-4663-ba9a-389e0b237d29","Type":"ContainerStarted","Data":"21f26df1940c27eaa29319d39c391798b35ed979f587394aeba8f839d2edb8a1"}
Oct 29 22:18:13.490112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:13.490079    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:13.922338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:13.922249    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:14.073877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:14.073846    4860 generic.go:356] "Generic (PLEG): container finished" podID="f0ad8058-a52d-4663-ba9a-389e0b237d29" containerID="b893af5b0baa87bc49a6e04c45023b8c841707c67f5b4e4ebc6ce9ae76b8ba27" exitCode=0
Oct 29 22:18:14.074339 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:14.073894    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg" event={"ID":"f0ad8058-a52d-4663-ba9a-389e0b237d29","Type":"ContainerDied","Data":"b893af5b0baa87bc49a6e04c45023b8c841707c67f5b4e4ebc6ce9ae76b8ba27"}
Oct 29 22:18:14.409900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:14.409869    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:14.520656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:14.520613    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:15.080077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:15.080039    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv" event={"ID":"0e42bac5-f252-4c5f-9739-f0b300489d73","Type":"ContainerStarted","Data":"67c6d61c1e582b607cf420dd31871a8a803dd946bc6fe936885334ae2e9738e4"}
Oct 29 22:18:15.080487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:15.080185    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"
Oct 29 22:18:15.122288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:15.122240    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv" podStartSLOduration=1.6756653099999999 podStartE2EDuration="4.122226642s" totalImagesPullingTime="2.446561332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:15.12202619 +0000 UTC m=+237.146802348" watchObservedRunningTime="2026-10-29 22:18:15.122226642 +0000 UTC m=+237.147002772"
Oct 29 22:18:16.086206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:16.086170    4860 generic.go:356] "Generic (PLEG): container finished" podID="f0ad8058-a52d-4663-ba9a-389e0b237d29" containerID="e43a6d20b10ed33c6faa292f7d127614691e07f15c3d37ed7b5d89fb6f5fb5dd" exitCode=0
Oct 29 22:18:16.086792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:16.086253    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg" event={"ID":"f0ad8058-a52d-4663-ba9a-389e0b237d29","Type":"ContainerDied","Data":"e43a6d20b10ed33c6faa292f7d127614691e07f15c3d37ed7b5d89fb6f5fb5dd"}
Oct 29 22:18:16.112872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:16.112848    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:16.228465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:16.228438    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:17.091711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:17.091673    4860 generic.go:356] "Generic (PLEG): container finished" podID="f0ad8058-a52d-4663-ba9a-389e0b237d29" containerID="8d51cca47c8f89f8d8e59ac59abd11231160a1f254358c6850a72ec8c4cdb168" exitCode=0
Oct 29 22:18:17.092103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:17.091747    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg" event={"ID":"f0ad8058-a52d-4663-ba9a-389e0b237d29","Type":"ContainerDied","Data":"8d51cca47c8f89f8d8e59ac59abd11231160a1f254358c6850a72ec8c4cdb168"}
Oct 29 22:18:18.053117 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:18:18.053083    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:18:18.219985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:18.219962    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:18.366232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:18.366160    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-bundle\") pod \"f0ad8058-a52d-4663-ba9a-389e0b237d29\" (UID: \"f0ad8058-a52d-4663-ba9a-389e0b237d29\") "
Oct 29 22:18:18.366396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:18.366235    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ad8058-a52d-4663-ba9a-389e0b237d29-kube-api-access-pxdzn\" (UniqueName: \"kubernetes.io/projected/f0ad8058-a52d-4663-ba9a-389e0b237d29-kube-api-access-pxdzn\") pod \"f0ad8058-a52d-4663-ba9a-389e0b237d29\" (UID: \"f0ad8058-a52d-4663-ba9a-389e0b237d29\") "
Oct 29 22:18:18.366396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:18.366261    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-util\" (UniqueName: \"kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-util\") pod \"f0ad8058-a52d-4663-ba9a-389e0b237d29\" (UID: \"f0ad8058-a52d-4663-ba9a-389e0b237d29\") "
Oct 29 22:18:18.366630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:18.366603    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-bundle" pod "f0ad8058-a52d-4663-ba9a-389e0b237d29" (UID: "f0ad8058-a52d-4663-ba9a-389e0b237d29"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:18:18.368359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:18.368312    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ad8058-a52d-4663-ba9a-389e0b237d29-kube-api-access-pxdzn" pod "f0ad8058-a52d-4663-ba9a-389e0b237d29" (UID: "f0ad8058-a52d-4663-ba9a-389e0b237d29"). InnerVolumeSpecName "kube-api-access-pxdzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:18:18.371514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:18.371493    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-util" pod "f0ad8058-a52d-4663-ba9a-389e0b237d29" (UID: "f0ad8058-a52d-4663-ba9a-389e0b237d29"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:18:18.467219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:18.467197    4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:18:18.467309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:18.467218    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxdzn\" (UniqueName: \"kubernetes.io/projected/f0ad8058-a52d-4663-ba9a-389e0b237d29-kube-api-access-pxdzn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:18:18.467309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:18.467234    4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0ad8058-a52d-4663-ba9a-389e0b237d29-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:18:19.100424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:19.100394    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg" event={"ID":"f0ad8058-a52d-4663-ba9a-389e0b237d29","Type":"ContainerDied","Data":"21f26df1940c27eaa29319d39c391798b35ed979f587394aeba8f839d2edb8a1"}
Oct 29 22:18:19.100592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:19.100426    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85qvbsg"
Oct 29 22:18:19.100592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:19.100438    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21f26df1940c27eaa29319d39c391798b35ed979f587394aeba8f839d2edb8a1"
Oct 29 22:18:19.496384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:19.496352    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:19.606798 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:19.606772    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:19.995364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:19.995311    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:20.117337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:20.117305    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:20.741602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:20.741559    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:21.088756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.088680    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv"
Oct 29 22:18:21.516652 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.516621    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-nrpqc"]
Oct 29 22:18:21.516971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.516954    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f0ad8058-a52d-4663-ba9a-389e0b237d29" containerName="pull"
Oct 29 22:18:21.517109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.516974    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ad8058-a52d-4663-ba9a-389e0b237d29" containerName="pull"
Oct 29 22:18:21.517109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.516994    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f0ad8058-a52d-4663-ba9a-389e0b237d29" containerName="util"
Oct 29 22:18:21.517109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.517001    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ad8058-a52d-4663-ba9a-389e0b237d29" containerName="util"
Oct 29 22:18:21.517109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.517031    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f0ad8058-a52d-4663-ba9a-389e0b237d29" containerName="extract"
Oct 29 22:18:21.517109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.517039    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ad8058-a52d-4663-ba9a-389e0b237d29" containerName="extract"
Oct 29 22:18:21.517417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.517121    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ad8058-a52d-4663-ba9a-389e0b237d29" containerName="extract"
Oct 29 22:18:21.519057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.519038    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-nrpqc"
Oct 29 22:18:21.542077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.542054    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-nrpqc"]
Oct 29 22:18:21.554186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.554165    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4zwj8\""
Oct 29 22:18:21.591269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.591243    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmzmk\" (UniqueName: \"kubernetes.io/projected/85cad816-1565-4f4d-85a5-8d8619fe175f-kube-api-access-tmzmk\") pod \"cert-manager-597b95cb84-nrpqc\" (UID: \"85cad816-1565-4f4d-85a5-8d8619fe175f\") " pod="cert-manager/cert-manager-597b95cb84-nrpqc"
Oct 29 22:18:21.591387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.591303    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85cad816-1565-4f4d-85a5-8d8619fe175f-bound-sa-token\") pod \"cert-manager-597b95cb84-nrpqc\" (UID: \"85cad816-1565-4f4d-85a5-8d8619fe175f\") " pod="cert-manager/cert-manager-597b95cb84-nrpqc"
Oct 29 22:18:21.692101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.692078    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85cad816-1565-4f4d-85a5-8d8619fe175f-bound-sa-token\") pod \"cert-manager-597b95cb84-nrpqc\" (UID: \"85cad816-1565-4f4d-85a5-8d8619fe175f\") " pod="cert-manager/cert-manager-597b95cb84-nrpqc"
Oct 29 22:18:21.692181 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.692114    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmzmk\" (UniqueName: \"kubernetes.io/projected/85cad816-1565-4f4d-85a5-8d8619fe175f-kube-api-access-tmzmk\") pod \"cert-manager-597b95cb84-nrpqc\" (UID: \"85cad816-1565-4f4d-85a5-8d8619fe175f\") " pod="cert-manager/cert-manager-597b95cb84-nrpqc"
Oct 29 22:18:21.712679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.712648    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85cad816-1565-4f4d-85a5-8d8619fe175f-bound-sa-token\") pod \"cert-manager-597b95cb84-nrpqc\" (UID: \"85cad816-1565-4f4d-85a5-8d8619fe175f\") " pod="cert-manager/cert-manager-597b95cb84-nrpqc"
Oct 29 22:18:21.712940 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.712917    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmzmk\" (UniqueName: \"kubernetes.io/projected/85cad816-1565-4f4d-85a5-8d8619fe175f-kube-api-access-tmzmk\") pod \"cert-manager-597b95cb84-nrpqc\" (UID: \"85cad816-1565-4f4d-85a5-8d8619fe175f\") " pod="cert-manager/cert-manager-597b95cb84-nrpqc"
Oct 29 22:18:21.829153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:21.829076    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-nrpqc"
Oct 29 22:18:22.119282 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:18:22.119252    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85cad816_1565_4f4d_85a5_8d8619fe175f.slice/crio-5386431d1788059e74fae6df97e602c21447c3b6e4d9d6b5f8a0aa1312cb16cf WatchSource:0}: Error finding container 5386431d1788059e74fae6df97e602c21447c3b6e4d9d6b5f8a0aa1312cb16cf: Status 404 returned error can't find the container with id 5386431d1788059e74fae6df97e602c21447c3b6e4d9d6b5f8a0aa1312cb16cf
Oct 29 22:18:22.119840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:22.119816    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-nrpqc"]
Oct 29 22:18:22.921663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:22.921626    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:23.031115 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:23.031084    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:23.115764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:23.115737    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-nrpqc" event={"ID":"85cad816-1565-4f4d-85a5-8d8619fe175f","Type":"ContainerStarted","Data":"54da79914add7feb78c19e9bbdacf4603f370aa7437de46743b71f64e7a9de59"}
Oct 29 22:18:23.115886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:23.115770    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-nrpqc" event={"ID":"85cad816-1565-4f4d-85a5-8d8619fe175f","Type":"ContainerStarted","Data":"5386431d1788059e74fae6df97e602c21447c3b6e4d9d6b5f8a0aa1312cb16cf"}
Oct 29 22:18:23.156143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:23.156105    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-nrpqc" podStartSLOduration=2.156092539 podStartE2EDuration="2.156092539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:23.155934081 +0000 UTC m=+245.180710223" watchObservedRunningTime="2026-10-29 22:18:23.156092539 +0000 UTC m=+245.180868667"
Oct 29 22:18:24.874468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:24.874439    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:25.009550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:25.009518    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:29.157010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:29.156977    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:29.277829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:29.277798    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:31.080436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.080406    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:31.288183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.288151    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"]
Oct 29 22:18:31.291481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.291465    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:31.307893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.307868    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 29 22:18:31.321849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.321824    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-9rrkz\""
Oct 29 22:18:31.321946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.321853    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 29 22:18:31.323154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.323135    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"]
Oct 29 22:18:31.354787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.354726    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t\" (UID: \"717fdb12-1d8a-4400-93e9-a201e09cc184\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:31.354787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.354765    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t\" (UID: \"717fdb12-1d8a-4400-93e9-a201e09cc184\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:31.354787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.354785    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvn68\" (UniqueName: \"kubernetes.io/projected/717fdb12-1d8a-4400-93e9-a201e09cc184-kube-api-access-fvn68\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t\" (UID: \"717fdb12-1d8a-4400-93e9-a201e09cc184\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:31.455988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.455966    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t\" (UID: \"717fdb12-1d8a-4400-93e9-a201e09cc184\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:31.456080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.456002    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t\" (UID: \"717fdb12-1d8a-4400-93e9-a201e09cc184\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:31.456080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.456020    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvn68\" (UniqueName: \"kubernetes.io/projected/717fdb12-1d8a-4400-93e9-a201e09cc184-kube-api-access-fvn68\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t\" (UID: \"717fdb12-1d8a-4400-93e9-a201e09cc184\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:31.456353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.456316    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t\" (UID: \"717fdb12-1d8a-4400-93e9-a201e09cc184\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:31.456395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.456365    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t\" (UID: \"717fdb12-1d8a-4400-93e9-a201e09cc184\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:31.476792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.476770    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvn68\" (UniqueName: \"kubernetes.io/projected/717fdb12-1d8a-4400-93e9-a201e09cc184-kube-api-access-fvn68\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t\" (UID: \"717fdb12-1d8a-4400-93e9-a201e09cc184\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:31.607436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.607386    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:31.899952 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:18:31.899914    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717fdb12_1d8a_4400_93e9_a201e09cc184.slice/crio-1e76e91fa665e240428c136bd3a18599301369630ab2b5ed2d5747b241d33baa WatchSource:0}: Error finding container 1e76e91fa665e240428c136bd3a18599301369630ab2b5ed2d5747b241d33baa: Status 404 returned error can't find the container with id 1e76e91fa665e240428c136bd3a18599301369630ab2b5ed2d5747b241d33baa
Oct 29 22:18:31.900540 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:31.900473    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"]
Oct 29 22:18:32.152949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:32.152880    4860 generic.go:356] "Generic (PLEG): container finished" podID="717fdb12-1d8a-4400-93e9-a201e09cc184" containerID="d295a06176d4c2e239385e978a90d70db01cb7fc6fd36fcc48bfad9d14ff85f7" exitCode=0
Oct 29 22:18:32.153273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:32.152958    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t" event={"ID":"717fdb12-1d8a-4400-93e9-a201e09cc184","Type":"ContainerDied","Data":"d295a06176d4c2e239385e978a90d70db01cb7fc6fd36fcc48bfad9d14ff85f7"}
Oct 29 22:18:32.153273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:32.153004    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t" event={"ID":"717fdb12-1d8a-4400-93e9-a201e09cc184","Type":"ContainerStarted","Data":"1e76e91fa665e240428c136bd3a18599301369630ab2b5ed2d5747b241d33baa"}
Oct 29 22:18:33.158570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:33.158486    4860 generic.go:356] "Generic (PLEG): container finished" podID="717fdb12-1d8a-4400-93e9-a201e09cc184" containerID="4e4ab6a529494a5bf12d57291a41a648ea7a9b39f008bb56bb6fee34b40e5584" exitCode=0
Oct 29 22:18:33.158938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:33.158578    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t" event={"ID":"717fdb12-1d8a-4400-93e9-a201e09cc184","Type":"ContainerDied","Data":"4e4ab6a529494a5bf12d57291a41a648ea7a9b39f008bb56bb6fee34b40e5584"}
Oct 29 22:18:34.164803 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:34.164763    4860 generic.go:356] "Generic (PLEG): container finished" podID="717fdb12-1d8a-4400-93e9-a201e09cc184" containerID="b7ac502025dd3a6efd09d1ac5335994a74baed1e72b50a2bd90a79ff12358d7b" exitCode=0
Oct 29 22:18:34.165161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:34.164841    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t" event={"ID":"717fdb12-1d8a-4400-93e9-a201e09cc184","Type":"ContainerDied","Data":"b7ac502025dd3a6efd09d1ac5335994a74baed1e72b50a2bd90a79ff12358d7b"}
Oct 29 22:18:35.287525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:35.287505    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:35.385585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:35.385561    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717fdb12-1d8a-4400-93e9-a201e09cc184-kube-api-access-fvn68\" (UniqueName: \"kubernetes.io/projected/717fdb12-1d8a-4400-93e9-a201e09cc184-kube-api-access-fvn68\") pod \"717fdb12-1d8a-4400-93e9-a201e09cc184\" (UID: \"717fdb12-1d8a-4400-93e9-a201e09cc184\") "
Oct 29 22:18:35.385728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:35.385599    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-util\" (UniqueName: \"kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-util\") pod \"717fdb12-1d8a-4400-93e9-a201e09cc184\" (UID: \"717fdb12-1d8a-4400-93e9-a201e09cc184\") "
Oct 29 22:18:35.385728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:35.385675    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-bundle\" (UniqueName: \"kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-bundle\") pod \"717fdb12-1d8a-4400-93e9-a201e09cc184\" (UID: \"717fdb12-1d8a-4400-93e9-a201e09cc184\") "
Oct 29 22:18:35.386274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:35.386240    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-bundle" pod "717fdb12-1d8a-4400-93e9-a201e09cc184" (UID: "717fdb12-1d8a-4400-93e9-a201e09cc184"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:18:35.387767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:35.387746    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717fdb12-1d8a-4400-93e9-a201e09cc184-kube-api-access-fvn68" pod "717fdb12-1d8a-4400-93e9-a201e09cc184" (UID: "717fdb12-1d8a-4400-93e9-a201e09cc184"). InnerVolumeSpecName "kube-api-access-fvn68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:18:35.391104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:35.391080    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-util" pod "717fdb12-1d8a-4400-93e9-a201e09cc184" (UID: "717fdb12-1d8a-4400-93e9-a201e09cc184"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:18:35.486789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:35.486759    4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:18:35.486789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:35.486785    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvn68\" (UniqueName: \"kubernetes.io/projected/717fdb12-1d8a-4400-93e9-a201e09cc184-kube-api-access-fvn68\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:18:35.486969 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:35.486797    4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/717fdb12-1d8a-4400-93e9-a201e09cc184-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:18:36.175207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:36.175178    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t"
Oct 29 22:18:36.175207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:36.175190    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlq8t" event={"ID":"717fdb12-1d8a-4400-93e9-a201e09cc184","Type":"ContainerDied","Data":"1e76e91fa665e240428c136bd3a18599301369630ab2b5ed2d5747b241d33baa"}
Oct 29 22:18:36.175447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:36.175228    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e76e91fa665e240428c136bd3a18599301369630ab2b5ed2d5747b241d33baa"
Oct 29 22:18:43.122334 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:43.122278    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:43.231840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:43.231809    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:43.858444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:43.858411    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:43.989623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:43.989595    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"]
Oct 29 22:18:43.989886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:43.989874    4860 cpu_manager.go:414] "containerMap: removing container" podUID="717fdb12-1d8a-4400-93e9-a201e09cc184" containerName="util"
Oct 29 22:18:43.989928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:43.989887    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717fdb12-1d8a-4400-93e9-a201e09cc184" containerName="util"
Oct 29 22:18:43.989928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:43.989903    4860 cpu_manager.go:414] "containerMap: removing container" podUID="717fdb12-1d8a-4400-93e9-a201e09cc184" containerName="pull"
Oct 29 22:18:43.989928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:43.989907    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717fdb12-1d8a-4400-93e9-a201e09cc184" containerName="pull"
Oct 29 22:18:43.989928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:43.989924    4860 cpu_manager.go:414] "containerMap: removing container" podUID="717fdb12-1d8a-4400-93e9-a201e09cc184" containerName="extract"
Oct 29 22:18:43.989928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:43.989929    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717fdb12-1d8a-4400-93e9-a201e09cc184" containerName="extract"
Oct 29 22:18:43.990074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:43.989998    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="717fdb12-1d8a-4400-93e9-a201e09cc184" containerName="extract"
Oct 29 22:18:43.993630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:43.993615    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.009996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.009972    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 29 22:18:44.010107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.009969    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 29 22:18:44.010107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.010017    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 29 22:18:44.010107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.010060    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 29 22:18:44.010107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.010097    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9jtb2\""
Oct 29 22:18:44.010253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.010162    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 29 22:18:44.032106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.032084    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"]
Oct 29 22:18:44.143704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.143636    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.143704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.143681    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.144054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.143720    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.144054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.143822    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnrvz\" (UniqueName: \"kubernetes.io/projected/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-kube-api-access-wnrvz\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.144054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.143854    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-manifests\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.144054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.143875    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-tmp\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.187970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.187936    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:44.244666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.244644    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.244785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.244681    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrvz\" (UniqueName: \"kubernetes.io/projected/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-kube-api-access-wnrvz\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.244785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.244702    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-manifests\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.244785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.244716    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-tmp\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.244785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.244741    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.244785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.244785    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.245207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.245176    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-manifests\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.247032 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.247003    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-tmp\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.247156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.247140    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.247229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.247153    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.247357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.247304    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.266095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.266071    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrvz\" (UniqueName: \"kubernetes.io/projected/6e67bb2d-9995-49bb-ba2c-00950cc17aaa-kube-api-access-wnrvz\") pod \"opendatahub-operator-controller-manager-6f7c578db4-f4vnd\" (UID: \"6e67bb2d-9995-49bb-ba2c-00950cc17aaa\") " pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.310016 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.309992    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:44.366172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.366131    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:44.495180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.495147    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:44.651176 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:18:44.651140    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e67bb2d_9995_49bb_ba2c_00950cc17aaa.slice/crio-b5d093cda954e8ce8a531941164c2218d149b85537aac65ec00e92fa19054aea WatchSource:0}: Error finding container b5d093cda954e8ce8a531941164c2218d149b85537aac65ec00e92fa19054aea: Status 404 returned error can't find the container with id b5d093cda954e8ce8a531941164c2218d149b85537aac65ec00e92fa19054aea
Oct 29 22:18:44.651472 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:44.651451    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"]
Oct 29 22:18:45.216919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:45.216879    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd" event={"ID":"6e67bb2d-9995-49bb-ba2c-00950cc17aaa","Type":"ContainerStarted","Data":"b5d093cda954e8ce8a531941164c2218d149b85537aac65ec00e92fa19054aea"}
Oct 29 22:18:47.542928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:47.542902    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:47.695164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:47.695141    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:48.052603 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:18:48.052572    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:18:48.231897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:48.231814    4860 generic.go:356] "Generic (PLEG): container finished" podID="6e67bb2d-9995-49bb-ba2c-00950cc17aaa" containerID="a40557989dd95d9b534af50220e3668c4c1b9a1c878ee61dd7a095964c359dc0" exitCode=0
Oct 29 22:18:48.232027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:48.231905    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd" event={"ID":"6e67bb2d-9995-49bb-ba2c-00950cc17aaa","Type":"ContainerDied","Data":"a40557989dd95d9b534af50220e3668c4c1b9a1c878ee61dd7a095964c359dc0"}
Oct 29 22:18:49.238265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:49.238227    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd" event={"ID":"6e67bb2d-9995-49bb-ba2c-00950cc17aaa","Type":"ContainerStarted","Data":"2744cb8ac35ad54a25a1ac0afe72bdf67daac0a07b3f78655908d78aa195e2cc"}
Oct 29 22:18:49.238670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:49.238346    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:18:49.325234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:49.325185    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd" podStartSLOduration=2.713333485 podStartE2EDuration="6.325174035s" totalImagesPullingTime="3.203767829s" totalInitContainerRuntime="408.072721ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:18:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:18:49.310525298 +0000 UTC m=+271.335301474" watchObservedRunningTime="2026-10-29 22:18:49.325174035 +0000 UTC m=+271.349950165"
Oct 29 22:18:57.582563 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:57.582524    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:18:58.572396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:18:58.572361    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:00.244449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:00.244417    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f7c578db4-f4vnd"
Oct 29 22:19:02.744552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:02.744521    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:02.906459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:02.906430    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:03.360754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.360719    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"]
Oct 29 22:19:03.364120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.364105    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"
Oct 29 22:19:03.395506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.395472    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:03.395839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.395814    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 29 22:19:03.395955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.395825    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 29 22:19:03.395955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.395869    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-tjdnx\""
Oct 29 22:19:03.398163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.398140    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"]
Oct 29 22:19:03.488039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.488017    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e-tmp\") pod \"kube-auth-proxy-7b84b87b7c-vx6bc\" (UID: \"59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"
Oct 29 22:19:03.488149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.488052    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e-tls-certs\") pod \"kube-auth-proxy-7b84b87b7c-vx6bc\" (UID: \"59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"
Oct 29 22:19:03.488149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.488093    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fxkl\" (UniqueName: \"kubernetes.io/projected/59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e-kube-api-access-5fxkl\") pod \"kube-auth-proxy-7b84b87b7c-vx6bc\" (UID: \"59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"
Oct 29 22:19:03.565613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.565579    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:03.588759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.588730    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fxkl\" (UniqueName: \"kubernetes.io/projected/59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e-kube-api-access-5fxkl\") pod \"kube-auth-proxy-7b84b87b7c-vx6bc\" (UID: \"59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"
Oct 29 22:19:03.588872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.588780    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e-tmp\") pod \"kube-auth-proxy-7b84b87b7c-vx6bc\" (UID: \"59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"
Oct 29 22:19:03.588872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.588824    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e-tls-certs\") pod \"kube-auth-proxy-7b84b87b7c-vx6bc\" (UID: \"59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"
Oct 29 22:19:03.591080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.591060    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e-tmp\") pod \"kube-auth-proxy-7b84b87b7c-vx6bc\" (UID: \"59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"
Oct 29 22:19:03.591264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.591246    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e-tls-certs\") pod \"kube-auth-proxy-7b84b87b7c-vx6bc\" (UID: \"59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"
Oct 29 22:19:03.609594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.609568    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fxkl\" (UniqueName: \"kubernetes.io/projected/59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e-kube-api-access-5fxkl\") pod \"kube-auth-proxy-7b84b87b7c-vx6bc\" (UID: \"59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e\") " pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"
Oct 29 22:19:03.676425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.676365    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"
Oct 29 22:19:03.967656 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:19:03.967625    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ddbdc4_f0f7_4d14_ac0b_83dca35e4e8e.slice/crio-e40a173c103fc036c380eb6a2aa0652f145b2173cdfcc6c2051ca5586648fbf2 WatchSource:0}: Error finding container e40a173c103fc036c380eb6a2aa0652f145b2173cdfcc6c2051ca5586648fbf2: Status 404 returned error can't find the container with id e40a173c103fc036c380eb6a2aa0652f145b2173cdfcc6c2051ca5586648fbf2
Oct 29 22:19:03.967998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:03.967935    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc"]
Oct 29 22:19:04.044941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.044915    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"]
Oct 29 22:19:04.049194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.049176    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.076728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.076704    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"]
Oct 29 22:19:04.146406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.146379    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:19:04.146602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.146388    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 29 22:19:04.146602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.146440    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 29 22:19:04.146768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.146476    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 29 22:19:04.146768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.146535    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-b922r\""
Oct 29 22:19:04.146768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.146636    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 29 22:19:04.194924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.194897    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a-metrics-cert\") pod \"lws-controller-manager-5b88c549dd-qpstw\" (UID: \"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.195041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.194935    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bvv\" (UniqueName: \"kubernetes.io/projected/72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a-kube-api-access-82bvv\") pod \"lws-controller-manager-5b88c549dd-qpstw\" (UID: \"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.195041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.194957    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a-manager-config\") pod \"lws-controller-manager-5b88c549dd-qpstw\" (UID: \"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.195131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.195063    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a-cert\") pod \"lws-controller-manager-5b88c549dd-qpstw\" (UID: \"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.295648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.295572    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a-metrics-cert\") pod \"lws-controller-manager-5b88c549dd-qpstw\" (UID: \"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.295648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.295608    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bvv\" (UniqueName: \"kubernetes.io/projected/72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a-kube-api-access-82bvv\") pod \"lws-controller-manager-5b88c549dd-qpstw\" (UID: \"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.295845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.295751    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a-manager-config\") pod \"lws-controller-manager-5b88c549dd-qpstw\" (UID: \"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.295845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.295820    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a-cert\") pod \"lws-controller-manager-5b88c549dd-qpstw\" (UID: \"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.296388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.296365    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a-manager-config\") pod \"lws-controller-manager-5b88c549dd-qpstw\" (UID: \"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.297969 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.297946    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a-cert\") pod \"lws-controller-manager-5b88c549dd-qpstw\" (UID: \"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.298055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.298038    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a-metrics-cert\") pod \"lws-controller-manager-5b88c549dd-qpstw\" (UID: \"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.304422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.304400    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc" event={"ID":"59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e","Type":"ContainerStarted","Data":"e40a173c103fc036c380eb6a2aa0652f145b2173cdfcc6c2051ca5586648fbf2"}
Oct 29 22:19:04.317729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.317706    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bvv\" (UniqueName: \"kubernetes.io/projected/72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a-kube-api-access-82bvv\") pod \"lws-controller-manager-5b88c549dd-qpstw\" (UID: \"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a\") " pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.361729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.361708    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:04.664306 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.664225    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"]
Oct 29 22:19:04.664306 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:19:04.664271    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ab07c0_3b73_4b7f_aefc_9ae8ec87ec3a.slice/crio-6df1ed1c566a40faa53b84b7ee1ed1f23af865918dec2cf097fe6ac073d7dec9 WatchSource:0}: Error finding container 6df1ed1c566a40faa53b84b7ee1ed1f23af865918dec2cf097fe6ac073d7dec9: Status 404 returned error can't find the container with id 6df1ed1c566a40faa53b84b7ee1ed1f23af865918dec2cf097fe6ac073d7dec9
Oct 29 22:19:04.667696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.667300    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:04.789461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:04.789423    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:05.311145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:05.311103    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw" event={"ID":"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a","Type":"ContainerStarted","Data":"6df1ed1c566a40faa53b84b7ee1ed1f23af865918dec2cf097fe6ac073d7dec9"}
Oct 29 22:19:05.627137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:05.627059    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:06.286458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:06.286402    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:06.318410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:06.318378    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw" event={"ID":"72ab07c0-3b73-4b7f-aefc-9ae8ec87ec3a","Type":"ContainerStarted","Data":"a0b54a60e1edf2f1e8e84433b138ea68bb0191c249294ef91874fa60768ca393"}
Oct 29 22:19:06.318766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:06.318511    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:06.360273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:06.360225    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw" podStartSLOduration=0.87031653 podStartE2EDuration="2.360212458s" totalImagesPullingTime="1.489895928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:06.359227402 +0000 UTC m=+288.384003530" watchObservedRunningTime="2026-10-29 22:19:06.360212458 +0000 UTC m=+288.384988589"
Oct 29 22:19:06.615755 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:06.615723    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:06.903162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:06.903085    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:07.324569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:07.324524    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc" event={"ID":"59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e","Type":"ContainerStarted","Data":"2641ac3d09301c412dca5970865a76d9c4a4c1675ae3db4c1b48bc043eab80f0"}
Oct 29 22:19:07.365086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:07.365041    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b84b87b7c-vx6bc" podStartSLOduration=1.880119428 podStartE2EDuration="4.365030743s" totalImagesPullingTime="2.484911315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:19:07.364593332 +0000 UTC m=+289.389369462" watchObservedRunningTime="2026-10-29 22:19:07.365030743 +0000 UTC m=+289.389806911"
Oct 29 22:19:08.559242 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:08.559208    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:08.667418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:08.667382    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:08.937172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:08.937145    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:09.055702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:09.055671    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:09.167657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:09.167625    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:09.308722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:09.308643    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:17.326895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:17.326854    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b88c549dd-qpstw"
Oct 29 22:19:18.053130 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:19:18.053095    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:19:18.134166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:18.134128    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:19:18.134959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:18.134934    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:19:19.720198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:19.720142    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:19.876730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:19.876702    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:32.879985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:19:32.879893    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:19:48.053767 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:19:48.053730    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:20:06.821677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:06.821632    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:06.965787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:06.965755    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:07.090937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:07.090862    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:07.235706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:07.235669    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:07.346167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:07.346096    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:07.601645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:07.601571    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:09.252905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:09.252872    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:09.362337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:09.362295    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:09.478478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:09.478452    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:09.598587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:09.598519    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:09.708193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:09.708168    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:09.814614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:09.814584    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:18.053822 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:20:18.053780    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:20:48.053629 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:20:48.053594    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:20:56.019042 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:56.019005    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:20:56.207068 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:20:56.207038    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:01.102150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:01.102074    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:09.365164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.365132    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:09.638544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.638474    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"]
Oct 29 22:21:09.642084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.642066    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:09.672927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.672900    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 29 22:21:09.673039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.672962    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 29 22:21:09.673093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.673060    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-lqprt\""
Oct 29 22:21:09.674581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.674561    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"]
Oct 29 22:21:09.709108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.709080    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv\" (UID: \"0e226d31-0e4c-4108-b472-6fb804e1c0df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:09.709211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.709121    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k74j2\" (UniqueName: \"kubernetes.io/projected/0e226d31-0e4c-4108-b472-6fb804e1c0df-kube-api-access-k74j2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv\" (UID: \"0e226d31-0e4c-4108-b472-6fb804e1c0df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:09.709264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.709245    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv\" (UID: \"0e226d31-0e4c-4108-b472-6fb804e1c0df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:09.731690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.731665    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"]
Oct 29 22:21:09.734906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.734888    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:09.752334 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.752304    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"]
Oct 29 22:21:09.761254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.761236    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-7qjsq\""
Oct 29 22:21:09.810078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.810057    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv\" (UID: \"0e226d31-0e4c-4108-b472-6fb804e1c0df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:09.810160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.810086    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k74j2\" (UniqueName: \"kubernetes.io/projected/0e226d31-0e4c-4108-b472-6fb804e1c0df-kube-api-access-k74j2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv\" (UID: \"0e226d31-0e4c-4108-b472-6fb804e1c0df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:09.810160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.810112    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln\" (UID: \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:09.810252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.810180    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv\" (UID: \"0e226d31-0e4c-4108-b472-6fb804e1c0df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:09.810252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.810211    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln\" (UID: \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:09.810252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.810237    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5758\" (UniqueName: \"kubernetes.io/projected/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-kube-api-access-z5758\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln\" (UID: \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:09.810463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.810440    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv\" (UID: \"0e226d31-0e4c-4108-b472-6fb804e1c0df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:09.810567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.810550    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv\" (UID: \"0e226d31-0e4c-4108-b472-6fb804e1c0df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:09.833974 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.833946    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k74j2\" (UniqueName: \"kubernetes.io/projected/0e226d31-0e4c-4108-b472-6fb804e1c0df-kube-api-access-k74j2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv\" (UID: \"0e226d31-0e4c-4108-b472-6fb804e1c0df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:09.835940 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.835922    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"]
Oct 29 22:21:09.839497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.839483    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:09.857094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.857073    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"]
Oct 29 22:21:09.861469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.861450    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-rbxhs\""
Oct 29 22:21:09.911203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.911156    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj\" (UID: \"9d5c085d-c4ef-4925-90c1-36290f939881\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:09.911203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.911187    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln\" (UID: \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:09.911354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.911204    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj\" (UID: \"9d5c085d-c4ef-4925-90c1-36290f939881\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:09.911354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.911232    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptvzq\" (UniqueName: \"kubernetes.io/projected/9d5c085d-c4ef-4925-90c1-36290f939881-kube-api-access-ptvzq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj\" (UID: \"9d5c085d-c4ef-4925-90c1-36290f939881\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:09.911468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.911444    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln\" (UID: \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:09.911503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.911488    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5758\" (UniqueName: \"kubernetes.io/projected/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-kube-api-access-z5758\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln\" (UID: \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:09.911657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.911637    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln\" (UID: \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:09.911710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.911696    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln\" (UID: \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:09.935192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.935170    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"]
Oct 29 22:21:09.937464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.937442    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5758\" (UniqueName: \"kubernetes.io/projected/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-kube-api-access-z5758\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln\" (UID: \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:09.938575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.938561    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:09.956560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.956541    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"]
Oct 29 22:21:09.957940 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.957919    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:09.962133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:09.962105    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-qdt4l\""
Oct 29 22:21:10.012747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.012719    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj\" (UID: \"9d5c085d-c4ef-4925-90c1-36290f939881\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:10.012868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.012757    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj\" (UID: \"9d5c085d-c4ef-4925-90c1-36290f939881\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:10.012868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.012792    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptvzq\" (UniqueName: \"kubernetes.io/projected/9d5c085d-c4ef-4925-90c1-36290f939881-kube-api-access-ptvzq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj\" (UID: \"9d5c085d-c4ef-4925-90c1-36290f939881\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:10.012868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.012811    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx\" (UID: \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:10.012868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.012826    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx\" (UID: \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:10.012868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.012845    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lmvt\" (UniqueName: \"kubernetes.io/projected/30e47f5a-50a5-4334-9c22-02f10f4ccb53-kube-api-access-9lmvt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx\" (UID: \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:10.013319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.013179    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj\" (UID: \"9d5c085d-c4ef-4925-90c1-36290f939881\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:10.013955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.013435    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj\" (UID: \"9d5c085d-c4ef-4925-90c1-36290f939881\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:10.036850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.033714    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptvzq\" (UniqueName: \"kubernetes.io/projected/9d5c085d-c4ef-4925-90c1-36290f939881-kube-api-access-ptvzq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj\" (UID: \"9d5c085d-c4ef-4925-90c1-36290f939881\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:10.050667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.050643    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:10.113582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.113553    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx\" (UID: \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:10.113697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.113586    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx\" (UID: \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:10.113697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.113608    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lmvt\" (UniqueName: \"kubernetes.io/projected/30e47f5a-50a5-4334-9c22-02f10f4ccb53-kube-api-access-9lmvt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx\" (UID: \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:10.114067 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.114039    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx\" (UID: \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:10.114452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.114281    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx\" (UID: \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:10.134562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.134540    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lmvt\" (UniqueName: \"kubernetes.io/projected/30e47f5a-50a5-4334-9c22-02f10f4ccb53-kube-api-access-9lmvt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx\" (UID: \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:10.157248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.157224    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:10.254274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.254250    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"]
Oct 29 22:21:10.256211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:21:10.256192    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e226d31_0e4c_4108_b472_6fb804e1c0df.slice/crio-a749ff6d663b9aadca16bb1c0aa910d4ea956b15f6ffa67f040e57e0b9a477b0 WatchSource:0}: Error finding container a749ff6d663b9aadca16bb1c0aa910d4ea956b15f6ffa67f040e57e0b9a477b0: Status 404 returned error can't find the container with id a749ff6d663b9aadca16bb1c0aa910d4ea956b15f6ffa67f040e57e0b9a477b0
Oct 29 22:21:10.261771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.261751    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:10.347935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.347907    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"]
Oct 29 22:21:10.450704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.450675    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"]
Oct 29 22:21:10.456460 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:21:10.456431    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5c085d_c4ef_4925_90c1_36290f939881.slice/crio-cff384dcfef668405008b8b31d2fe39cb741e43747d6dfd477ea287505f2f905 WatchSource:0}: Error finding container cff384dcfef668405008b8b31d2fe39cb741e43747d6dfd477ea287505f2f905: Status 404 returned error can't find the container with id cff384dcfef668405008b8b31d2fe39cb741e43747d6dfd477ea287505f2f905
Oct 29 22:21:10.559908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.559880    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"]
Oct 29 22:21:10.607105 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:21:10.607080    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e47f5a_50a5_4334_9c22_02f10f4ccb53.slice/crio-1c67afcd7d046e0cef7b828728b42ff0b526f9e993b09f7b08b42e3c4cda70a7 WatchSource:0}: Error finding container 1c67afcd7d046e0cef7b828728b42ff0b526f9e993b09f7b08b42e3c4cda70a7: Status 404 returned error can't find the container with id 1c67afcd7d046e0cef7b828728b42ff0b526f9e993b09f7b08b42e3c4cda70a7
Oct 29 22:21:10.854900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.854864    4860 generic.go:356] "Generic (PLEG): container finished" podID="ed439a2b-6992-4138-9394-b9ffc5d2a0c7" containerID="be501fe91252106f1f1951ac7ce76214566fb92ba339ba626d484bb509039856" exitCode=0
Oct 29 22:21:10.855010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.854948    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln" event={"ID":"ed439a2b-6992-4138-9394-b9ffc5d2a0c7","Type":"ContainerDied","Data":"be501fe91252106f1f1951ac7ce76214566fb92ba339ba626d484bb509039856"}
Oct 29 22:21:10.855010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.854999    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln" event={"ID":"ed439a2b-6992-4138-9394-b9ffc5d2a0c7","Type":"ContainerStarted","Data":"2fba0287f4b0645b8f390365150a14326673a76c6cd24ba09469d0c24987efea"}
Oct 29 22:21:10.856170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.856154    4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:21:10.856695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.856672    4860 generic.go:356] "Generic (PLEG): container finished" podID="30e47f5a-50a5-4334-9c22-02f10f4ccb53" containerID="2b24a7a375b9376160034014c154572f18c4344e3adde7dabc9b9681a1331da3" exitCode=0
Oct 29 22:21:10.856799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.856750    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx" event={"ID":"30e47f5a-50a5-4334-9c22-02f10f4ccb53","Type":"ContainerDied","Data":"2b24a7a375b9376160034014c154572f18c4344e3adde7dabc9b9681a1331da3"}
Oct 29 22:21:10.856867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.856800    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx" event={"ID":"30e47f5a-50a5-4334-9c22-02f10f4ccb53","Type":"ContainerStarted","Data":"1c67afcd7d046e0cef7b828728b42ff0b526f9e993b09f7b08b42e3c4cda70a7"}
Oct 29 22:21:10.858447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.858398    4860 generic.go:356] "Generic (PLEG): container finished" podID="0e226d31-0e4c-4108-b472-6fb804e1c0df" containerID="4abfbc23a85b3e85051d7d4ef3b7b7c0bfa916a5fcf163f8474db1847c07c5e3" exitCode=0
Oct 29 22:21:10.858521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.858442    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv" event={"ID":"0e226d31-0e4c-4108-b472-6fb804e1c0df","Type":"ContainerDied","Data":"4abfbc23a85b3e85051d7d4ef3b7b7c0bfa916a5fcf163f8474db1847c07c5e3"}
Oct 29 22:21:10.858521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.858500    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv" event={"ID":"0e226d31-0e4c-4108-b472-6fb804e1c0df","Type":"ContainerStarted","Data":"a749ff6d663b9aadca16bb1c0aa910d4ea956b15f6ffa67f040e57e0b9a477b0"}
Oct 29 22:21:10.860047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.860025    4860 generic.go:356] "Generic (PLEG): container finished" podID="9d5c085d-c4ef-4925-90c1-36290f939881" containerID="ac8301b1945cb07a5d1a28446723894b65a711b3a732de44170e4fc017d48865" exitCode=0
Oct 29 22:21:10.860146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.860070    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj" event={"ID":"9d5c085d-c4ef-4925-90c1-36290f939881","Type":"ContainerDied","Data":"ac8301b1945cb07a5d1a28446723894b65a711b3a732de44170e4fc017d48865"}
Oct 29 22:21:10.860146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:10.860118    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj" event={"ID":"9d5c085d-c4ef-4925-90c1-36290f939881","Type":"ContainerStarted","Data":"cff384dcfef668405008b8b31d2fe39cb741e43747d6dfd477ea287505f2f905"}
Oct 29 22:21:11.866454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:11.866423    4860 generic.go:356] "Generic (PLEG): container finished" podID="ed439a2b-6992-4138-9394-b9ffc5d2a0c7" containerID="c740c288e1e2fa75b400e8d14fdc78b86e458db90265485f7879d9f55b09882c" exitCode=0
Oct 29 22:21:11.866825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:11.866513    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln" event={"ID":"ed439a2b-6992-4138-9394-b9ffc5d2a0c7","Type":"ContainerDied","Data":"c740c288e1e2fa75b400e8d14fdc78b86e458db90265485f7879d9f55b09882c"}
Oct 29 22:21:11.868692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:11.868664    4860 generic.go:356] "Generic (PLEG): container finished" podID="30e47f5a-50a5-4334-9c22-02f10f4ccb53" containerID="9059b65fc1064142091c2a312dd0a41a335825abe44c8d2a3a78064e74271eb0" exitCode=0
Oct 29 22:21:11.868799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:11.868692    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx" event={"ID":"30e47f5a-50a5-4334-9c22-02f10f4ccb53","Type":"ContainerDied","Data":"9059b65fc1064142091c2a312dd0a41a335825abe44c8d2a3a78064e74271eb0"}
Oct 29 22:21:11.870591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:11.870573    4860 generic.go:356] "Generic (PLEG): container finished" podID="0e226d31-0e4c-4108-b472-6fb804e1c0df" containerID="eb66d3f5fe4259f4f3dde1a58798ed9046acd3a22b3712c0f106f3d705fe1af0" exitCode=0
Oct 29 22:21:11.870673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:11.870606    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv" event={"ID":"0e226d31-0e4c-4108-b472-6fb804e1c0df","Type":"ContainerDied","Data":"eb66d3f5fe4259f4f3dde1a58798ed9046acd3a22b3712c0f106f3d705fe1af0"}
Oct 29 22:21:12.877180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:12.877151    4860 generic.go:356] "Generic (PLEG): container finished" podID="9d5c085d-c4ef-4925-90c1-36290f939881" containerID="72059b418d231fbe071df740e7d7114b84dce8f9b726fd841293b7d62a4d1e56" exitCode=0
Oct 29 22:21:12.877558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:12.877236    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj" event={"ID":"9d5c085d-c4ef-4925-90c1-36290f939881","Type":"ContainerDied","Data":"72059b418d231fbe071df740e7d7114b84dce8f9b726fd841293b7d62a4d1e56"}
Oct 29 22:21:12.879705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:12.879681    4860 generic.go:356] "Generic (PLEG): container finished" podID="ed439a2b-6992-4138-9394-b9ffc5d2a0c7" containerID="7917a1503c55f29797cd6e9f9bf9e0c89438e5a02760edec4a8304074a7a7097" exitCode=0
Oct 29 22:21:12.879800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:12.879776    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln" event={"ID":"ed439a2b-6992-4138-9394-b9ffc5d2a0c7","Type":"ContainerDied","Data":"7917a1503c55f29797cd6e9f9bf9e0c89438e5a02760edec4a8304074a7a7097"}
Oct 29 22:21:12.881826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:12.881802    4860 generic.go:356] "Generic (PLEG): container finished" podID="30e47f5a-50a5-4334-9c22-02f10f4ccb53" containerID="0ea18d6316dc5ce4ea4799413696e10b16fbab4d1aaedf68dcada95453f8e7f1" exitCode=0
Oct 29 22:21:12.881927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:12.881874    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx" event={"ID":"30e47f5a-50a5-4334-9c22-02f10f4ccb53","Type":"ContainerDied","Data":"0ea18d6316dc5ce4ea4799413696e10b16fbab4d1aaedf68dcada95453f8e7f1"}
Oct 29 22:21:12.883880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:12.883859    4860 generic.go:356] "Generic (PLEG): container finished" podID="0e226d31-0e4c-4108-b472-6fb804e1c0df" containerID="fc2a8d92110af9e108eb09562e5a676a82b1ecd62adbf0139339e2d7dc6059d3" exitCode=0
Oct 29 22:21:12.883981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:12.883899    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv" event={"ID":"0e226d31-0e4c-4108-b472-6fb804e1c0df","Type":"ContainerDied","Data":"fc2a8d92110af9e108eb09562e5a676a82b1ecd62adbf0139339e2d7dc6059d3"}
Oct 29 22:21:13.891470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:13.891436    4860 generic.go:356] "Generic (PLEG): container finished" podID="9d5c085d-c4ef-4925-90c1-36290f939881" containerID="a1322739cb5e1134c557f972e24eec429cef6e8c43bb6627bbebfad39e715ee2" exitCode=0
Oct 29 22:21:13.891866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:13.891515    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj" event={"ID":"9d5c085d-c4ef-4925-90c1-36290f939881","Type":"ContainerDied","Data":"a1322739cb5e1134c557f972e24eec429cef6e8c43bb6627bbebfad39e715ee2"}
Oct 29 22:21:14.051727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.051702    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:14.079714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.079691    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:14.083011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.082991    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:14.147079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.146983    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e226d31-0e4c-4108-b472-6fb804e1c0df-kube-api-access-k74j2\" (UniqueName: \"kubernetes.io/projected/0e226d31-0e4c-4108-b472-6fb804e1c0df-kube-api-access-k74j2\") pod \"0e226d31-0e4c-4108-b472-6fb804e1c0df\" (UID: \"0e226d31-0e4c-4108-b472-6fb804e1c0df\") "
Oct 29 22:21:14.147079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.147047    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-bundle\" (UniqueName: \"kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-bundle\") pod \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\" (UID: \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\") "
Oct 29 22:21:14.147294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.147101    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-bundle\") pod \"0e226d31-0e4c-4108-b472-6fb804e1c0df\" (UID: \"0e226d31-0e4c-4108-b472-6fb804e1c0df\") "
Oct 29 22:21:14.147294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.147136    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e47f5a-50a5-4334-9c22-02f10f4ccb53-kube-api-access-9lmvt\" (UniqueName: \"kubernetes.io/projected/30e47f5a-50a5-4334-9c22-02f10f4ccb53-kube-api-access-9lmvt\") pod \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\" (UID: \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\") "
Oct 29 22:21:14.147294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.147167    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-kube-api-access-z5758\" (UniqueName: \"kubernetes.io/projected/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-kube-api-access-z5758\") pod \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\" (UID: \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\") "
Oct 29 22:21:14.147294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.147202    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-util\" (UniqueName: \"kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-util\") pod \"0e226d31-0e4c-4108-b472-6fb804e1c0df\" (UID: \"0e226d31-0e4c-4108-b472-6fb804e1c0df\") "
Oct 29 22:21:14.147294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.147228    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-util\" (UniqueName: \"kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-util\") pod \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\" (UID: \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\") "
Oct 29 22:21:14.147294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.147262    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-util\" (UniqueName: \"kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-util\") pod \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\" (UID: \"30e47f5a-50a5-4334-9c22-02f10f4ccb53\") "
Oct 29 22:21:14.148263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.148225    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-bundle" pod "30e47f5a-50a5-4334-9c22-02f10f4ccb53" (UID: "30e47f5a-50a5-4334-9c22-02f10f4ccb53"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:21:14.148387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.148302    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-bundle\") pod \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\" (UID: \"ed439a2b-6992-4138-9394-b9ffc5d2a0c7\") "
Oct 29 22:21:14.148845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.148818    4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:21:14.150357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.149497    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-bundle" pod "0e226d31-0e4c-4108-b472-6fb804e1c0df" (UID: "0e226d31-0e4c-4108-b472-6fb804e1c0df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:21:14.150357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.149685    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-bundle" pod "ed439a2b-6992-4138-9394-b9ffc5d2a0c7" (UID: "ed439a2b-6992-4138-9394-b9ffc5d2a0c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:21:14.150504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.150471    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-kube-api-access-z5758" pod "ed439a2b-6992-4138-9394-b9ffc5d2a0c7" (UID: "ed439a2b-6992-4138-9394-b9ffc5d2a0c7"). InnerVolumeSpecName "kube-api-access-z5758". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:21:14.150936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.150890    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e47f5a-50a5-4334-9c22-02f10f4ccb53-kube-api-access-9lmvt" pod "30e47f5a-50a5-4334-9c22-02f10f4ccb53" (UID: "30e47f5a-50a5-4334-9c22-02f10f4ccb53"). InnerVolumeSpecName "kube-api-access-9lmvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:21:14.152689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.152664    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e226d31-0e4c-4108-b472-6fb804e1c0df-kube-api-access-k74j2" pod "0e226d31-0e4c-4108-b472-6fb804e1c0df" (UID: "0e226d31-0e4c-4108-b472-6fb804e1c0df"). InnerVolumeSpecName "kube-api-access-k74j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:21:14.154858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.154824    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-util" pod "30e47f5a-50a5-4334-9c22-02f10f4ccb53" (UID: "30e47f5a-50a5-4334-9c22-02f10f4ccb53"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:21:14.157650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.157604    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-util" pod "ed439a2b-6992-4138-9394-b9ffc5d2a0c7" (UID: "ed439a2b-6992-4138-9394-b9ffc5d2a0c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:21:14.160119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.160099    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-util" pod "0e226d31-0e4c-4108-b472-6fb804e1c0df" (UID: "0e226d31-0e4c-4108-b472-6fb804e1c0df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:21:14.249899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.249874    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k74j2\" (UniqueName: \"kubernetes.io/projected/0e226d31-0e4c-4108-b472-6fb804e1c0df-kube-api-access-k74j2\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:21:14.249899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.249898    4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:21:14.250079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.249915    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lmvt\" (UniqueName: \"kubernetes.io/projected/30e47f5a-50a5-4334-9c22-02f10f4ccb53-kube-api-access-9lmvt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:21:14.250079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.249928    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5758\" (UniqueName: \"kubernetes.io/projected/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-kube-api-access-z5758\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:21:14.250079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.249940    4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e226d31-0e4c-4108-b472-6fb804e1c0df-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:21:14.250079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.249952    4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:21:14.250079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.249963    4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30e47f5a-50a5-4334-9c22-02f10f4ccb53-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:21:14.250079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.249974    4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed439a2b-6992-4138-9394-b9ffc5d2a0c7-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:21:14.898232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.898200    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln"
Oct 29 22:21:14.898232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.898212    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zq9ln" event={"ID":"ed439a2b-6992-4138-9394-b9ffc5d2a0c7","Type":"ContainerDied","Data":"2fba0287f4b0645b8f390365150a14326673a76c6cd24ba09469d0c24987efea"}
Oct 29 22:21:14.898702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.898260    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fba0287f4b0645b8f390365150a14326673a76c6cd24ba09469d0c24987efea"
Oct 29 22:21:14.900278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.900256    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx"
Oct 29 22:21:14.900407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.900250    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pm2gx" event={"ID":"30e47f5a-50a5-4334-9c22-02f10f4ccb53","Type":"ContainerDied","Data":"1c67afcd7d046e0cef7b828728b42ff0b526f9e993b09f7b08b42e3c4cda70a7"}
Oct 29 22:21:14.900407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.900372    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c67afcd7d046e0cef7b828728b42ff0b526f9e993b09f7b08b42e3c4cda70a7"
Oct 29 22:21:14.902202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.902182    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv"
Oct 29 22:21:14.902291 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.902197    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cx44lv" event={"ID":"0e226d31-0e4c-4108-b472-6fb804e1c0df","Type":"ContainerDied","Data":"a749ff6d663b9aadca16bb1c0aa910d4ea956b15f6ffa67f040e57e0b9a477b0"}
Oct 29 22:21:14.902291 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:14.902236    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a749ff6d663b9aadca16bb1c0aa910d4ea956b15f6ffa67f040e57e0b9a477b0"
Oct 29 22:21:15.013823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.013801    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:15.055087 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.055043    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-bundle\") pod \"9d5c085d-c4ef-4925-90c1-36290f939881\" (UID: \"9d5c085d-c4ef-4925-90c1-36290f939881\") "
Oct 29 22:21:15.055248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.055143    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-util\" (UniqueName: \"kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-util\") pod \"9d5c085d-c4ef-4925-90c1-36290f939881\" (UID: \"9d5c085d-c4ef-4925-90c1-36290f939881\") "
Oct 29 22:21:15.055248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.055195    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5c085d-c4ef-4925-90c1-36290f939881-kube-api-access-ptvzq\" (UniqueName: \"kubernetes.io/projected/9d5c085d-c4ef-4925-90c1-36290f939881-kube-api-access-ptvzq\") pod \"9d5c085d-c4ef-4925-90c1-36290f939881\" (UID: \"9d5c085d-c4ef-4925-90c1-36290f939881\") "
Oct 29 22:21:15.056360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.056301    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-bundle" pod "9d5c085d-c4ef-4925-90c1-36290f939881" (UID: "9d5c085d-c4ef-4925-90c1-36290f939881"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:21:15.057838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.057809    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5c085d-c4ef-4925-90c1-36290f939881-kube-api-access-ptvzq" pod "9d5c085d-c4ef-4925-90c1-36290f939881" (UID: "9d5c085d-c4ef-4925-90c1-36290f939881"). InnerVolumeSpecName "kube-api-access-ptvzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:21:15.061941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.061915    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-util" pod "9d5c085d-c4ef-4925-90c1-36290f939881" (UID: "9d5c085d-c4ef-4925-90c1-36290f939881"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:21:15.155903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.155834    4860 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-util\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:21:15.155903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.155861    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptvzq\" (UniqueName: \"kubernetes.io/projected/9d5c085d-c4ef-4925-90c1-36290f939881-kube-api-access-ptvzq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:21:15.155903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.155871    4860 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d5c085d-c4ef-4925-90c1-36290f939881-bundle\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:21:15.908081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.908053    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj"
Oct 29 22:21:15.908455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.908056    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aljxhj" event={"ID":"9d5c085d-c4ef-4925-90c1-36290f939881","Type":"ContainerDied","Data":"cff384dcfef668405008b8b31d2fe39cb741e43747d6dfd477ea287505f2f905"}
Oct 29 22:21:15.908455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:15.908170    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff384dcfef668405008b8b31d2fe39cb741e43747d6dfd477ea287505f2f905"
Oct 29 22:21:18.052981 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:21:18.052943    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:21:22.832745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:22.832709    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:23.631946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:23.631915    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:26.843946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:26.843891    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:27.066182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:27.066153    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:27.183280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:27.183240    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:28.474316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:28.474273    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:29.241811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:29.241778    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:29.881377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:29.881345    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:31.232664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:31.232634    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:31.659402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:31.659293    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:32.056261 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:32.056227    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:32.568505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:32.568474    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:32.694121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:32.694093    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:36.643176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:36.643137    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:36.773790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:36.773758    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:37.253876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:37.253848    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:37.376226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:37.376195    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:39.640399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:39.640363    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:39.773056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:39.773028    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:42.631292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631259    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"]
Oct 29 22:21:42.631751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631729    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9d5c085d-c4ef-4925-90c1-36290f939881" containerName="util"
Oct 29 22:21:42.631799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631755    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5c085d-c4ef-4925-90c1-36290f939881" containerName="util"
Oct 29 22:21:42.631799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631771    4860 cpu_manager.go:414] "containerMap: removing container" podUID="30e47f5a-50a5-4334-9c22-02f10f4ccb53" containerName="pull"
Oct 29 22:21:42.631799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631781    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e47f5a-50a5-4334-9c22-02f10f4ccb53" containerName="pull"
Oct 29 22:21:42.631799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631791    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0e226d31-0e4c-4108-b472-6fb804e1c0df" containerName="pull"
Oct 29 22:21:42.631799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631798    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e226d31-0e4c-4108-b472-6fb804e1c0df" containerName="pull"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631819    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ed439a2b-6992-4138-9394-b9ffc5d2a0c7" containerName="util"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631827    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed439a2b-6992-4138-9394-b9ffc5d2a0c7" containerName="util"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631837    4860 cpu_manager.go:414] "containerMap: removing container" podUID="30e47f5a-50a5-4334-9c22-02f10f4ccb53" containerName="extract"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631844    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e47f5a-50a5-4334-9c22-02f10f4ccb53" containerName="extract"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631855    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ed439a2b-6992-4138-9394-b9ffc5d2a0c7" containerName="pull"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631861    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed439a2b-6992-4138-9394-b9ffc5d2a0c7" containerName="pull"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631881    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9d5c085d-c4ef-4925-90c1-36290f939881" containerName="extract"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631886    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5c085d-c4ef-4925-90c1-36290f939881" containerName="extract"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631899    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0e226d31-0e4c-4108-b472-6fb804e1c0df" containerName="util"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631904    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e226d31-0e4c-4108-b472-6fb804e1c0df" containerName="util"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631912    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ed439a2b-6992-4138-9394-b9ffc5d2a0c7" containerName="extract"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631919    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed439a2b-6992-4138-9394-b9ffc5d2a0c7" containerName="extract"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631931    4860 cpu_manager.go:414] "containerMap: removing container" podUID="30e47f5a-50a5-4334-9c22-02f10f4ccb53" containerName="util"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631938    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e47f5a-50a5-4334-9c22-02f10f4ccb53" containerName="util"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631945    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9d5c085d-c4ef-4925-90c1-36290f939881" containerName="pull"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631949    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5c085d-c4ef-4925-90c1-36290f939881" containerName="pull"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631957    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0e226d31-0e4c-4108-b472-6fb804e1c0df" containerName="extract"
Oct 29 22:21:42.631966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.631961    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e226d31-0e4c-4108-b472-6fb804e1c0df" containerName="extract"
Oct 29 22:21:42.632567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.632039    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e47f5a-50a5-4334-9c22-02f10f4ccb53" containerName="extract"
Oct 29 22:21:42.632567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.632049    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5c085d-c4ef-4925-90c1-36290f939881" containerName="extract"
Oct 29 22:21:42.632567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.632056    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed439a2b-6992-4138-9394-b9ffc5d2a0c7" containerName="extract"
Oct 29 22:21:42.632567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.632063    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e226d31-0e4c-4108-b472-6fb804e1c0df" containerName="extract"
Oct 29 22:21:42.635369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.635353    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"
Oct 29 22:21:42.664645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.664614    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:42.674170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.674131    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 29 22:21:42.674170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.674141    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 29 22:21:42.674170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.674143    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 29 22:21:42.674170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.674171    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 29 22:21:42.674476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.674175    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qwmm7\""
Oct 29 22:21:42.676059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.676035    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"]
Oct 29 22:21:42.769494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.769470    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c70a71db-8aee-4f26-b5ef-abbcfcaf4440-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-v9gbd\" (UID: \"c70a71db-8aee-4f26-b5ef-abbcfcaf4440\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"
Oct 29 22:21:42.769626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.769506    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c70a71db-8aee-4f26-b5ef-abbcfcaf4440-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-v9gbd\" (UID: \"c70a71db-8aee-4f26-b5ef-abbcfcaf4440\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"
Oct 29 22:21:42.769626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.769530    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx8wk\" (UniqueName: \"kubernetes.io/projected/c70a71db-8aee-4f26-b5ef-abbcfcaf4440-kube-api-access-jx8wk\") pod \"kuadrant-console-plugin-66c4fb575d-v9gbd\" (UID: \"c70a71db-8aee-4f26-b5ef-abbcfcaf4440\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"
Oct 29 22:21:42.870545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.870516    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c70a71db-8aee-4f26-b5ef-abbcfcaf4440-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-v9gbd\" (UID: \"c70a71db-8aee-4f26-b5ef-abbcfcaf4440\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"
Oct 29 22:21:42.870679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.870561    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c70a71db-8aee-4f26-b5ef-abbcfcaf4440-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-v9gbd\" (UID: \"c70a71db-8aee-4f26-b5ef-abbcfcaf4440\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"
Oct 29 22:21:42.870738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.870673    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8wk\" (UniqueName: \"kubernetes.io/projected/c70a71db-8aee-4f26-b5ef-abbcfcaf4440-kube-api-access-jx8wk\") pod \"kuadrant-console-plugin-66c4fb575d-v9gbd\" (UID: \"c70a71db-8aee-4f26-b5ef-abbcfcaf4440\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"
Oct 29 22:21:42.871215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.871194    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c70a71db-8aee-4f26-b5ef-abbcfcaf4440-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-v9gbd\" (UID: \"c70a71db-8aee-4f26-b5ef-abbcfcaf4440\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"
Oct 29 22:21:42.873061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.873039    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c70a71db-8aee-4f26-b5ef-abbcfcaf4440-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-v9gbd\" (UID: \"c70a71db-8aee-4f26-b5ef-abbcfcaf4440\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"
Oct 29 22:21:42.890232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.890168    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:42.891620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.891604    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8wk\" (UniqueName: \"kubernetes.io/projected/c70a71db-8aee-4f26-b5ef-abbcfcaf4440-kube-api-access-jx8wk\") pod \"kuadrant-console-plugin-66c4fb575d-v9gbd\" (UID: \"c70a71db-8aee-4f26-b5ef-abbcfcaf4440\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"
Oct 29 22:21:42.946641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:42.946613    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"
Oct 29 22:21:43.238239 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:21:43.238208    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc70a71db_8aee_4f26_b5ef_abbcfcaf4440.slice/crio-de4cd9afc0ba5608c988c21443c1510defda0ae4ae5482b61c2eb80b249d0039 WatchSource:0}: Error finding container de4cd9afc0ba5608c988c21443c1510defda0ae4ae5482b61c2eb80b249d0039: Status 404 returned error can't find the container with id de4cd9afc0ba5608c988c21443c1510defda0ae4ae5482b61c2eb80b249d0039
Oct 29 22:21:43.239745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:43.239720    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd"]
Oct 29 22:21:43.869163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:43.869120    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:43.984009 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:43.983978    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:44.042597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:44.042539    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd" event={"ID":"c70a71db-8aee-4f26-b5ef-abbcfcaf4440","Type":"ContainerStarted","Data":"de4cd9afc0ba5608c988c21443c1510defda0ae4ae5482b61c2eb80b249d0039"}
Oct 29 22:21:48.058456 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:21:48.058420    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:21:48.580471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:48.580439    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:48.699305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:48.699257    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:49.068981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:49.068945    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd" event={"ID":"c70a71db-8aee-4f26-b5ef-abbcfcaf4440","Type":"ContainerStarted","Data":"8b7b4ed4e32bf853ece3ab4c2a14a7c0ad13240b3b02be9bf8d71233c6fa6741"}
Oct 29 22:21:49.110806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:49.110763    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-v9gbd" podStartSLOduration=1.689465046 podStartE2EDuration="7.110750708s" totalImagesPullingTime="5.421285662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:49.109673924 +0000 UTC m=+451.134450054" watchObservedRunningTime="2026-10-29 22:21:49.110750708 +0000 UTC m=+451.135526836"
Oct 29 22:21:56.683505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:56.683470    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:56.807757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:56.807721    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:57.282658 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:57.282627    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:21:57.402303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:21:57.402275    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:00.269690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:00.269654    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:00.398143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:00.398090    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:04.880146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:04.880115    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:05.004115 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:05.004083    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:08.638938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:08.638900    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:08.761120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:08.761077    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:09.500050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:09.500012    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:18.053491 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:22:18.053453    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:22:26.370515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:26.370476    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:26.485679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:26.485648    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:30.868196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:30.868121    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:30.976422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:30.976394    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:40.099683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:40.099648    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:40.212568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:40.212538    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:40.321347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:40.321302    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:40.910820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:40.910787    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dtqbh"]
Oct 29 22:22:40.914174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:40.914150    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dtqbh"
Oct 29 22:22:40.929469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:40.929447    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dtqbh"]
Oct 29 22:22:40.938900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:40.938880    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-w47ht\""
Oct 29 22:22:40.992847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:40.992819    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jkn\" (UniqueName: \"kubernetes.io/projected/e44ee238-0cd2-4d8f-89f5-75287d11257c-kube-api-access-b7jkn\") pod \"authorino-57f7f8659f-dtqbh\" (UID: \"e44ee238-0cd2-4d8f-89f5-75287d11257c\") " pod="kuadrant-system/authorino-57f7f8659f-dtqbh"
Oct 29 22:22:41.008851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.008822    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:41.093904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.093883    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jkn\" (UniqueName: \"kubernetes.io/projected/e44ee238-0cd2-4d8f-89f5-75287d11257c-kube-api-access-b7jkn\") pod \"authorino-57f7f8659f-dtqbh\" (UID: \"e44ee238-0cd2-4d8f-89f5-75287d11257c\") " pod="kuadrant-system/authorino-57f7f8659f-dtqbh"
Oct 29 22:22:41.114149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.114121    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jkn\" (UniqueName: \"kubernetes.io/projected/e44ee238-0cd2-4d8f-89f5-75287d11257c-kube-api-access-b7jkn\") pod \"authorino-57f7f8659f-dtqbh\" (UID: \"e44ee238-0cd2-4d8f-89f5-75287d11257c\") " pod="kuadrant-system/authorino-57f7f8659f-dtqbh"
Oct 29 22:22:41.116034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.116009    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:22:41.225786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.225736    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dtqbh"
Oct 29 22:22:41.516534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.516466    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-k84dc"]
Oct 29 22:22:41.526808 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:22:41.526779    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode44ee238_0cd2_4d8f_89f5_75287d11257c.slice/crio-1c7a8cf1a2a6374205f67d7c4748b1a252e3ca334446a61128c17ed042722ea6 WatchSource:0}: Error finding container 1c7a8cf1a2a6374205f67d7c4748b1a252e3ca334446a61128c17ed042722ea6: Status 404 returned error can't find the container with id 1c7a8cf1a2a6374205f67d7c4748b1a252e3ca334446a61128c17ed042722ea6
Oct 29 22:22:41.527059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.527039    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dtqbh"]
Oct 29 22:22:41.527155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.527142    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-k84dc"
Oct 29 22:22:41.536102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.536079    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-k84dc"]
Oct 29 22:22:41.598228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.598202    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjl5r\" (UniqueName: \"kubernetes.io/projected/6d36307e-4caf-4071-a169-e3274026bf93-kube-api-access-hjl5r\") pod \"authorino-56c6c9ddd5-k84dc\" (UID: \"6d36307e-4caf-4071-a169-e3274026bf93\") " pod="kuadrant-system/authorino-56c6c9ddd5-k84dc"
Oct 29 22:22:41.699313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.699282    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjl5r\" (UniqueName: \"kubernetes.io/projected/6d36307e-4caf-4071-a169-e3274026bf93-kube-api-access-hjl5r\") pod \"authorino-56c6c9ddd5-k84dc\" (UID: \"6d36307e-4caf-4071-a169-e3274026bf93\") " pod="kuadrant-system/authorino-56c6c9ddd5-k84dc"
Oct 29 22:22:41.719606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.719580    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjl5r\" (UniqueName: \"kubernetes.io/projected/6d36307e-4caf-4071-a169-e3274026bf93-kube-api-access-hjl5r\") pod \"authorino-56c6c9ddd5-k84dc\" (UID: \"6d36307e-4caf-4071-a169-e3274026bf93\") " pod="kuadrant-system/authorino-56c6c9ddd5-k84dc"
Oct 29 22:22:41.845704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:41.845647    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-k84dc"
Oct 29 22:22:42.135424 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:22:42.135392    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d36307e_4caf_4071_a169_e3274026bf93.slice/crio-c6b37524ffd1ffabd08aa56e8cc3ca8c9cb9138d9ad79916bf8481b839cf2522 WatchSource:0}: Error finding container c6b37524ffd1ffabd08aa56e8cc3ca8c9cb9138d9ad79916bf8481b839cf2522: Status 404 returned error can't find the container with id c6b37524ffd1ffabd08aa56e8cc3ca8c9cb9138d9ad79916bf8481b839cf2522
Oct 29 22:22:42.135745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:42.135576    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-k84dc"]
Oct 29 22:22:42.319252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:42.319217    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-k84dc" event={"ID":"6d36307e-4caf-4071-a169-e3274026bf93","Type":"ContainerStarted","Data":"c6b37524ffd1ffabd08aa56e8cc3ca8c9cb9138d9ad79916bf8481b839cf2522"}
Oct 29 22:22:42.322028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:42.321985    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dtqbh" event={"ID":"e44ee238-0cd2-4d8f-89f5-75287d11257c","Type":"ContainerStarted","Data":"1c7a8cf1a2a6374205f67d7c4748b1a252e3ca334446a61128c17ed042722ea6"}
Oct 29 22:22:45.339519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:45.339484    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dtqbh" event={"ID":"e44ee238-0cd2-4d8f-89f5-75287d11257c","Type":"ContainerStarted","Data":"9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea"}
Oct 29 22:22:45.341023 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:45.340998    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-k84dc" event={"ID":"6d36307e-4caf-4071-a169-e3274026bf93","Type":"ContainerStarted","Data":"77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2"}
Oct 29 22:22:45.379573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:45.379532    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-dtqbh" podStartSLOduration=2.527627279 podStartE2EDuration="5.379520409s" totalImagesPullingTime="2.85189313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:45.378556857 +0000 UTC m=+507.403332987" watchObservedRunningTime="2026-10-29 22:22:45.379520409 +0000 UTC m=+507.404296591"
Oct 29 22:22:45.421082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:45.421036    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-k84dc" podStartSLOduration=1.817088396 podStartE2EDuration="4.421023304s" totalImagesPullingTime="2.603934908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:45.419372479 +0000 UTC m=+507.444148610" watchObservedRunningTime="2026-10-29 22:22:45.421023304 +0000 UTC m=+507.445799435"
Oct 29 22:22:45.452331 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:45.452286    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dtqbh"]
Oct 29 22:22:47.349797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:47.349747    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-dtqbh" podUID="e44ee238-0cd2-4d8f-89f5-75287d11257c" containerName="authorino" containerID="cri-o://9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea" gracePeriod=30
Oct 29 22:22:47.592272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:47.592251    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dtqbh"
Oct 29 22:22:47.650641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:47.650589    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44ee238-0cd2-4d8f-89f5-75287d11257c-kube-api-access-b7jkn\" (UniqueName: \"kubernetes.io/projected/e44ee238-0cd2-4d8f-89f5-75287d11257c-kube-api-access-b7jkn\") pod \"e44ee238-0cd2-4d8f-89f5-75287d11257c\" (UID: \"e44ee238-0cd2-4d8f-89f5-75287d11257c\") "
Oct 29 22:22:47.652684 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:47.652659    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44ee238-0cd2-4d8f-89f5-75287d11257c-kube-api-access-b7jkn" pod "e44ee238-0cd2-4d8f-89f5-75287d11257c" (UID: "e44ee238-0cd2-4d8f-89f5-75287d11257c"). InnerVolumeSpecName "kube-api-access-b7jkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:22:47.751675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:47.751651    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7jkn\" (UniqueName: \"kubernetes.io/projected/e44ee238-0cd2-4d8f-89f5-75287d11257c-kube-api-access-b7jkn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:22:48.052817 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:22:48.052782    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:22:48.355064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:48.354978    4860 generic.go:356] "Generic (PLEG): container finished" podID="e44ee238-0cd2-4d8f-89f5-75287d11257c" containerID="9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea" exitCode=0
Oct 29 22:22:48.355064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:48.355035    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-dtqbh"
Oct 29 22:22:48.355064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:48.355040    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dtqbh" event={"ID":"e44ee238-0cd2-4d8f-89f5-75287d11257c","Type":"ContainerDied","Data":"9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea"}
Oct 29 22:22:48.355577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:48.355087    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-dtqbh" event={"ID":"e44ee238-0cd2-4d8f-89f5-75287d11257c","Type":"ContainerDied","Data":"1c7a8cf1a2a6374205f67d7c4748b1a252e3ca334446a61128c17ed042722ea6"}
Oct 29 22:22:48.355577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:48.355105    4860 scope.go:122] "RemoveContainer" containerID="9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea"
Oct 29 22:22:48.363746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:48.363730    4860 scope.go:122] "RemoveContainer" containerID="9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea"
Oct 29 22:22:48.364022 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:22:48.364002    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea\": container with ID starting with 9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea not found: ID does not exist" containerID="9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea"
Oct 29 22:22:48.364072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:48.364031    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea"} err="failed to get container status \"9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea\": rpc error: code = NotFound desc = could not find container \"9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea\": container with ID starting with 9f3af8dc0b5352da431977d5fb9ffb7a66abeec1ed360687e332962be0b8eaea not found: ID does not exist"
Oct 29 22:22:48.408634 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:48.408606    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dtqbh"]
Oct 29 22:22:48.422554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:48.422533    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-dtqbh"]
Oct 29 22:22:50.109277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:22:50.109244    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44ee238-0cd2-4d8f-89f5-75287d11257c" path="/var/lib/kubelet/pods/e44ee238-0cd2-4d8f-89f5-75287d11257c/volumes"
Oct 29 22:23:00.916240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:00.916199    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:01.033158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:01.033129    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:01.679420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:01.679373    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:09.603582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:09.603539    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:14.971584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:14.971551    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:15.092361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.092313    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:15.277146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.277080    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r4tgz"]
Oct 29 22:23:15.277437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.277423    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e44ee238-0cd2-4d8f-89f5-75287d11257c" containerName="authorino"
Oct 29 22:23:15.277483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.277438    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44ee238-0cd2-4d8f-89f5-75287d11257c" containerName="authorino"
Oct 29 22:23:15.277517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.277504    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44ee238-0cd2-4d8f-89f5-75287d11257c" containerName="authorino"
Oct 29 22:23:15.281981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.281968    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-r4tgz"
Oct 29 22:23:15.297397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.297375    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r4tgz"]
Oct 29 22:23:15.438303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.438275    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r4tgz"]
Oct 29 22:23:15.438444 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:23:15.438427    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cl47w], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-r4tgz" podUID="a132e54d-ac59-48a6-a2f7-229d58322119"
Oct 29 22:23:15.459539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.459515    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cqw8m"]
Oct 29 22:23:15.461312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.461294    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl47w\" (UniqueName: \"kubernetes.io/projected/a132e54d-ac59-48a6-a2f7-229d58322119-kube-api-access-cl47w\") pod \"authorino-6f6944bcdd-r4tgz\" (UID: \"a132e54d-ac59-48a6-a2f7-229d58322119\") " pod="kuadrant-system/authorino-6f6944bcdd-r4tgz"
Oct 29 22:23:15.462568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.462550    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-cqw8m"
Oct 29 22:23:15.475537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.475517    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-r4tgz"
Oct 29 22:23:15.478507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.478488    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 29 22:23:15.479269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.479251    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cqw8m"]
Oct 29 22:23:15.482201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.482183    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-r4tgz"
Oct 29 22:23:15.562350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.562263    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl47w\" (UniqueName: \"kubernetes.io/projected/a132e54d-ac59-48a6-a2f7-229d58322119-kube-api-access-cl47w\") pod \"authorino-6f6944bcdd-r4tgz\" (UID: \"a132e54d-ac59-48a6-a2f7-229d58322119\") " pod="kuadrant-system/authorino-6f6944bcdd-r4tgz"
Oct 29 22:23:15.562350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.562300    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2b5238d6-715e-4db0-9882-34131e22e426-tls-cert\") pod \"authorino-6889cbd8c4-cqw8m\" (UID: \"2b5238d6-715e-4db0-9882-34131e22e426\") " pod="kuadrant-system/authorino-6889cbd8c4-cqw8m"
Oct 29 22:23:15.562498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.562374    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwz5c\" (UniqueName: \"kubernetes.io/projected/2b5238d6-715e-4db0-9882-34131e22e426-kube-api-access-qwz5c\") pod \"authorino-6889cbd8c4-cqw8m\" (UID: \"2b5238d6-715e-4db0-9882-34131e22e426\") " pod="kuadrant-system/authorino-6889cbd8c4-cqw8m"
Oct 29 22:23:15.582437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.582416    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl47w\" (UniqueName: \"kubernetes.io/projected/a132e54d-ac59-48a6-a2f7-229d58322119-kube-api-access-cl47w\") pod \"authorino-6f6944bcdd-r4tgz\" (UID: \"a132e54d-ac59-48a6-a2f7-229d58322119\") " pod="kuadrant-system/authorino-6f6944bcdd-r4tgz"
Oct 29 22:23:15.615178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.615148    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cqw8m"]
Oct 29 22:23:15.615375 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:23:15.615354    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qwz5c tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-cqw8m" podUID="2b5238d6-715e-4db0-9882-34131e22e426"
Oct 29 22:23:15.641011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.640984    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-955745748-t749f"]
Oct 29 22:23:15.644070 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.644055    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-955745748-t749f"
Oct 29 22:23:15.663223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.663202    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-955745748-t749f"]
Oct 29 22:23:15.663517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.663497    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2b5238d6-715e-4db0-9882-34131e22e426-tls-cert\") pod \"authorino-6889cbd8c4-cqw8m\" (UID: \"2b5238d6-715e-4db0-9882-34131e22e426\") " pod="kuadrant-system/authorino-6889cbd8c4-cqw8m"
Oct 29 22:23:15.663616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.663541    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwz5c\" (UniqueName: \"kubernetes.io/projected/2b5238d6-715e-4db0-9882-34131e22e426-kube-api-access-qwz5c\") pod \"authorino-6889cbd8c4-cqw8m\" (UID: \"2b5238d6-715e-4db0-9882-34131e22e426\") " pod="kuadrant-system/authorino-6889cbd8c4-cqw8m"
Oct 29 22:23:15.666030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.666011    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2b5238d6-715e-4db0-9882-34131e22e426-tls-cert\") pod \"authorino-6889cbd8c4-cqw8m\" (UID: \"2b5238d6-715e-4db0-9882-34131e22e426\") " pod="kuadrant-system/authorino-6889cbd8c4-cqw8m"
Oct 29 22:23:15.684006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.683981    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwz5c\" (UniqueName: \"kubernetes.io/projected/2b5238d6-715e-4db0-9882-34131e22e426-kube-api-access-qwz5c\") pod \"authorino-6889cbd8c4-cqw8m\" (UID: \"2b5238d6-715e-4db0-9882-34131e22e426\") " pod="kuadrant-system/authorino-6889cbd8c4-cqw8m"
Oct 29 22:23:15.764073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.764052    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a132e54d-ac59-48a6-a2f7-229d58322119-kube-api-access-cl47w\" (UniqueName: \"kubernetes.io/projected/a132e54d-ac59-48a6-a2f7-229d58322119-kube-api-access-cl47w\") pod \"a132e54d-ac59-48a6-a2f7-229d58322119\" (UID: \"a132e54d-ac59-48a6-a2f7-229d58322119\") "
Oct 29 22:23:15.764205 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.764193    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dbee6249-c27a-429b-836c-d9c1d9c86e6a-tls-cert\") pod \"authorino-955745748-t749f\" (UID: \"dbee6249-c27a-429b-836c-d9c1d9c86e6a\") " pod="kuadrant-system/authorino-955745748-t749f"
Oct 29 22:23:15.764250 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.764230    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks6gk\" (UniqueName: \"kubernetes.io/projected/dbee6249-c27a-429b-836c-d9c1d9c86e6a-kube-api-access-ks6gk\") pod \"authorino-955745748-t749f\" (UID: \"dbee6249-c27a-429b-836c-d9c1d9c86e6a\") " pod="kuadrant-system/authorino-955745748-t749f"
Oct 29 22:23:15.765957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.765934    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a132e54d-ac59-48a6-a2f7-229d58322119-kube-api-access-cl47w" pod "a132e54d-ac59-48a6-a2f7-229d58322119" (UID: "a132e54d-ac59-48a6-a2f7-229d58322119"). InnerVolumeSpecName "kube-api-access-cl47w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:23:15.865439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.865393    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dbee6249-c27a-429b-836c-d9c1d9c86e6a-tls-cert\") pod \"authorino-955745748-t749f\" (UID: \"dbee6249-c27a-429b-836c-d9c1d9c86e6a\") " pod="kuadrant-system/authorino-955745748-t749f"
Oct 29 22:23:15.865527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.865439    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6gk\" (UniqueName: \"kubernetes.io/projected/dbee6249-c27a-429b-836c-d9c1d9c86e6a-kube-api-access-ks6gk\") pod \"authorino-955745748-t749f\" (UID: \"dbee6249-c27a-429b-836c-d9c1d9c86e6a\") " pod="kuadrant-system/authorino-955745748-t749f"
Oct 29 22:23:15.865527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.865502    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl47w\" (UniqueName: \"kubernetes.io/projected/a132e54d-ac59-48a6-a2f7-229d58322119-kube-api-access-cl47w\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:23:15.867503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.867484    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dbee6249-c27a-429b-836c-d9c1d9c86e6a-tls-cert\") pod \"authorino-955745748-t749f\" (UID: \"dbee6249-c27a-429b-836c-d9c1d9c86e6a\") " pod="kuadrant-system/authorino-955745748-t749f"
Oct 29 22:23:15.885826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.885805    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6gk\" (UniqueName: \"kubernetes.io/projected/dbee6249-c27a-429b-836c-d9c1d9c86e6a-kube-api-access-ks6gk\") pod \"authorino-955745748-t749f\" (UID: \"dbee6249-c27a-429b-836c-d9c1d9c86e6a\") " pod="kuadrant-system/authorino-955745748-t749f"
Oct 29 22:23:15.955357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:15.955312    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-955745748-t749f"
Oct 29 22:23:16.245382 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:23:16.245354    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbee6249_c27a_429b_836c_d9c1d9c86e6a.slice/crio-d59b6beb2ba398475a5f7124bef3cca0328c8b995d0a1b0d9a198445c53660f6 WatchSource:0}: Error finding container d59b6beb2ba398475a5f7124bef3cca0328c8b995d0a1b0d9a198445c53660f6: Status 404 returned error can't find the container with id d59b6beb2ba398475a5f7124bef3cca0328c8b995d0a1b0d9a198445c53660f6
Oct 29 22:23:16.247282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.247254    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-955745748-t749f"]
Oct 29 22:23:16.481066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.481034    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-955745748-t749f" event={"ID":"dbee6249-c27a-429b-836c-d9c1d9c86e6a","Type":"ContainerStarted","Data":"d59b6beb2ba398475a5f7124bef3cca0328c8b995d0a1b0d9a198445c53660f6"}
Oct 29 22:23:16.481216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.481069    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-r4tgz"
Oct 29 22:23:16.481216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.481069    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-cqw8m"
Oct 29 22:23:16.489617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.489597    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-cqw8m"
Oct 29 22:23:16.597244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.597184    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r4tgz"]
Oct 29 22:23:16.599100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.599081    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-r4tgz"]
Oct 29 22:23:16.671147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.671125    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5238d6-715e-4db0-9882-34131e22e426-kube-api-access-qwz5c\" (UniqueName: \"kubernetes.io/projected/2b5238d6-715e-4db0-9882-34131e22e426-kube-api-access-qwz5c\") pod \"2b5238d6-715e-4db0-9882-34131e22e426\" (UID: \"2b5238d6-715e-4db0-9882-34131e22e426\") "
Oct 29 22:23:16.671264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.671179    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b5238d6-715e-4db0-9882-34131e22e426-tls-cert\" (UniqueName: \"kubernetes.io/secret/2b5238d6-715e-4db0-9882-34131e22e426-tls-cert\") pod \"2b5238d6-715e-4db0-9882-34131e22e426\" (UID: \"2b5238d6-715e-4db0-9882-34131e22e426\") "
Oct 29 22:23:16.673125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.673092    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b5238d6-715e-4db0-9882-34131e22e426-tls-cert" pod "2b5238d6-715e-4db0-9882-34131e22e426" (UID: "2b5238d6-715e-4db0-9882-34131e22e426"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:23:16.673230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.673219    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5238d6-715e-4db0-9882-34131e22e426-kube-api-access-qwz5c" pod "2b5238d6-715e-4db0-9882-34131e22e426" (UID: "2b5238d6-715e-4db0-9882-34131e22e426"). InnerVolumeSpecName "kube-api-access-qwz5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:23:16.771986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.771967    4860 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2b5238d6-715e-4db0-9882-34131e22e426-tls-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:23:16.771986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:16.771987    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwz5c\" (UniqueName: \"kubernetes.io/projected/2b5238d6-715e-4db0-9882-34131e22e426-kube-api-access-qwz5c\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:23:17.486724 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:17.486688    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-955745748-t749f" event={"ID":"dbee6249-c27a-429b-836c-d9c1d9c86e6a","Type":"ContainerStarted","Data":"a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09"}
Oct 29 22:23:17.487130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:17.486847    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-cqw8m"
Oct 29 22:23:17.527962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:17.527914    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-955745748-t749f" podStartSLOduration=2.052650544 podStartE2EDuration="2.527901751s" totalImagesPullingTime="475.251207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:17.527273233 +0000 UTC m=+539.552049364" watchObservedRunningTime="2026-10-29 22:23:17.527901751 +0000 UTC m=+539.552677881"
Oct 29 22:23:17.551509 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:17.551484    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-k84dc"]
Oct 29 22:23:17.551677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:17.551652    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-k84dc" podUID="6d36307e-4caf-4071-a169-e3274026bf93" containerName="authorino" containerID="cri-o://77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2" gracePeriod=30
Oct 29 22:23:17.587293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:17.587262    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cqw8m"]
Oct 29 22:23:17.588987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:17.588967    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cqw8m"]
Oct 29 22:23:17.791547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:17.791528    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-k84dc"
Oct 29 22:23:17.983574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:17.983545    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d36307e-4caf-4071-a169-e3274026bf93-kube-api-access-hjl5r\" (UniqueName: \"kubernetes.io/projected/6d36307e-4caf-4071-a169-e3274026bf93-kube-api-access-hjl5r\") pod \"6d36307e-4caf-4071-a169-e3274026bf93\" (UID: \"6d36307e-4caf-4071-a169-e3274026bf93\") "
Oct 29 22:23:17.985589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:17.985566    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d36307e-4caf-4071-a169-e3274026bf93-kube-api-access-hjl5r" pod "6d36307e-4caf-4071-a169-e3274026bf93" (UID: "6d36307e-4caf-4071-a169-e3274026bf93"). InnerVolumeSpecName "kube-api-access-hjl5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:23:18.052648 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:23:18.052571    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:23:18.084535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:18.084510    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjl5r\" (UniqueName: \"kubernetes.io/projected/6d36307e-4caf-4071-a169-e3274026bf93-kube-api-access-hjl5r\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:23:18.109229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:18.109206    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5238d6-715e-4db0-9882-34131e22e426" path="/var/lib/kubelet/pods/2b5238d6-715e-4db0-9882-34131e22e426/volumes"
Oct 29 22:23:18.109449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:18.109435    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a132e54d-ac59-48a6-a2f7-229d58322119" path="/var/lib/kubelet/pods/a132e54d-ac59-48a6-a2f7-229d58322119/volumes"
Oct 29 22:23:18.491987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:18.491955    4860 generic.go:356] "Generic (PLEG): container finished" podID="6d36307e-4caf-4071-a169-e3274026bf93" containerID="77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2" exitCode=0
Oct 29 22:23:18.492412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:18.492010    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-k84dc"
Oct 29 22:23:18.492412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:18.492049    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-k84dc" event={"ID":"6d36307e-4caf-4071-a169-e3274026bf93","Type":"ContainerDied","Data":"77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2"}
Oct 29 22:23:18.492412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:18.492085    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-k84dc" event={"ID":"6d36307e-4caf-4071-a169-e3274026bf93","Type":"ContainerDied","Data":"c6b37524ffd1ffabd08aa56e8cc3ca8c9cb9138d9ad79916bf8481b839cf2522"}
Oct 29 22:23:18.492412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:18.492102    4860 scope.go:122] "RemoveContainer" containerID="77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2"
Oct 29 22:23:18.500164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:18.500149    4860 scope.go:122] "RemoveContainer" containerID="77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2"
Oct 29 22:23:18.500450 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:23:18.500428    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2\": container with ID starting with 77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2 not found: ID does not exist" containerID="77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2"
Oct 29 22:23:18.500535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:18.500455    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2"} err="failed to get container status \"77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2\": rpc error: code = NotFound desc = could not find container \"77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2\": container with ID starting with 77c152226ef48e896fba45de531a3b4957f23893a88bdd701f1c36c88fccdcd2 not found: ID does not exist"
Oct 29 22:23:18.545452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:18.545432    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-k84dc"]
Oct 29 22:23:18.551059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:18.551040    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-k84dc"]
Oct 29 22:23:20.109408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:20.109368    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d36307e-4caf-4071-a169-e3274026bf93" path="/var/lib/kubelet/pods/6d36307e-4caf-4071-a169-e3274026bf93/volumes"
Oct 29 22:23:24.759148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:24.759115    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:24.878255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:24.878225    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:24.996968 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:24.996936    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:25.109932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:25.109866    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:28.778449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:28.778417    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:28.910215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:28.910170    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:32.118683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:32.118650    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:32.387816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:23:32.387737    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:23:48.053949 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:23:48.053908    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:24:00.190000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:00.189919    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:24:01.264914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:01.264883    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:24:01.894642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:01.894605    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:24:02.005294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:02.005264    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:24:02.122527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:02.122493    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:24:02.246718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:02.246685    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:24:09.718316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:09.718282    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:24:18.053678 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:24:18.053638    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:24:18.171571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:18.171549    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:24:18.175143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:18.175121    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:24:36.261068 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:36.261037    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:24:36.370972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:36.370943    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:24:46.759427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:46.759394    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:24:46.939111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:24:46.939078    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:24:48.053440 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:24:48.053405    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:25:04.866022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.865987    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"]
Oct 29 22:25:04.866633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.866500    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6d36307e-4caf-4071-a169-e3274026bf93" containerName="authorino"
Oct 29 22:25:04.866633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.866521    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d36307e-4caf-4071-a169-e3274026bf93" containerName="authorino"
Oct 29 22:25:04.866633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.866624    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d36307e-4caf-4071-a169-e3274026bf93" containerName="authorino"
Oct 29 22:25:04.869903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.869881    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:04.886203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.886180    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 29 22:25:04.888863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.888833    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"]
Oct 29 22:25:04.900102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.900078    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 29 22:25:04.900102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.900094    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 29 22:25:04.945606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.945580    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:04.986157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.986134    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-grkds\""
Oct 29 22:25:04.986299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.986169    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f34300ae-1300-4d57-af82-cb727c236caf-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:04.986299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.986243    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34300ae-1300-4d57-af82-cb727c236caf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:04.986441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.986318    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f34300ae-1300-4d57-af82-cb727c236caf-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:04.986441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.986422    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f34300ae-1300-4d57-af82-cb727c236caf-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:04.986519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.986444    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxqtd\" (UniqueName: \"kubernetes.io/projected/f34300ae-1300-4d57-af82-cb727c236caf-kube-api-access-rxqtd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:04.986519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:04.986462    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f34300ae-1300-4d57-af82-cb727c236caf-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.086770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.086742    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f34300ae-1300-4d57-af82-cb727c236caf-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.086885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.086773    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34300ae-1300-4d57-af82-cb727c236caf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.086885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.086799    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f34300ae-1300-4d57-af82-cb727c236caf-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.086885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.086859    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f34300ae-1300-4d57-af82-cb727c236caf-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.087020 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.086886    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxqtd\" (UniqueName: \"kubernetes.io/projected/f34300ae-1300-4d57-af82-cb727c236caf-kube-api-access-rxqtd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.087020 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.086911    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f34300ae-1300-4d57-af82-cb727c236caf-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.087275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.087248    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f34300ae-1300-4d57-af82-cb727c236caf-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.087367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.087273    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34300ae-1300-4d57-af82-cb727c236caf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.087367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.087308    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f34300ae-1300-4d57-af82-cb727c236caf-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.088976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.088953    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f34300ae-1300-4d57-af82-cb727c236caf-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.089150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.089130    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f34300ae-1300-4d57-af82-cb727c236caf-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.108985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.108957    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxqtd\" (UniqueName: \"kubernetes.io/projected/f34300ae-1300-4d57-af82-cb727c236caf-kube-api-access-rxqtd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2p75h\" (UID: \"f34300ae-1300-4d57-af82-cb727c236caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.183056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.183038    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:05.477452 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:25:05.477423    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34300ae_1300_4d57_af82_cb727c236caf.slice/crio-87578c5cc42171f71e674cc0911561cfef4f5bc44de17e7412746954582b8f51 WatchSource:0}: Error finding container 87578c5cc42171f71e674cc0911561cfef4f5bc44de17e7412746954582b8f51: Status 404 returned error can't find the container with id 87578c5cc42171f71e674cc0911561cfef4f5bc44de17e7412746954582b8f51
Oct 29 22:25:05.477924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.477901    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"]
Oct 29 22:25:05.976792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:05.976758    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h" event={"ID":"f34300ae-1300-4d57-af82-cb727c236caf","Type":"ContainerStarted","Data":"87578c5cc42171f71e674cc0911561cfef4f5bc44de17e7412746954582b8f51"}
Oct 29 22:25:07.990214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:07.990139    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h" event={"ID":"f34300ae-1300-4d57-af82-cb727c236caf","Type":"ContainerStarted","Data":"a501d15d870b13eb59dcb4339794eea03077c5e81a94b7495ef869cd9c4d0710"}
Oct 29 22:25:08.035239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.035179    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h" podStartSLOduration=1.790364335 podStartE2EDuration="4.035161685s" totalImagesPullingTime="2.24479735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:08.032723683 +0000 UTC m=+650.057499826" watchObservedRunningTime="2026-10-29 22:25:08.035161685 +0000 UTC m=+650.059937819"
Oct 29 22:25:08.184133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.184101    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:08.693010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.692978    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"]
Oct 29 22:25:08.696405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.696386    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.716183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.716160    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"]
Oct 29 22:25:08.727340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.727307    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 29 22:25:08.786056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.786033    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:08.835030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.835009    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da28494b-2ff0-4782-b53f-2583dc161b9d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.835134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.835046    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da28494b-2ff0-4782-b53f-2583dc161b9d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.835134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.835070    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvn6n\" (UniqueName: \"kubernetes.io/projected/da28494b-2ff0-4782-b53f-2583dc161b9d-kube-api-access-kvn6n\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.835134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.835119    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da28494b-2ff0-4782-b53f-2583dc161b9d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.835255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.835191    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da28494b-2ff0-4782-b53f-2583dc161b9d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.835255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.835220    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da28494b-2ff0-4782-b53f-2583dc161b9d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.935892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.935870    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da28494b-2ff0-4782-b53f-2583dc161b9d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.935979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.935907    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da28494b-2ff0-4782-b53f-2583dc161b9d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.935979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.935927    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvn6n\" (UniqueName: \"kubernetes.io/projected/da28494b-2ff0-4782-b53f-2583dc161b9d-kube-api-access-kvn6n\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.935979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.935945    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da28494b-2ff0-4782-b53f-2583dc161b9d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.936095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.935970    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da28494b-2ff0-4782-b53f-2583dc161b9d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.936095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.936047    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da28494b-2ff0-4782-b53f-2583dc161b9d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.936348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.936304    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da28494b-2ff0-4782-b53f-2583dc161b9d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.936424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.936382    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da28494b-2ff0-4782-b53f-2583dc161b9d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.936505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.936489    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da28494b-2ff0-4782-b53f-2583dc161b9d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.938220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.938203    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da28494b-2ff0-4782-b53f-2583dc161b9d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.938398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.938383    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da28494b-2ff0-4782-b53f-2583dc161b9d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:08.955938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:08.955887    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvn6n\" (UniqueName: \"kubernetes.io/projected/da28494b-2ff0-4782-b53f-2583dc161b9d-kube-api-access-kvn6n\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m\" (UID: \"da28494b-2ff0-4782-b53f-2583dc161b9d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:09.008254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:09.008226    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:09.300932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:09.300907    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"]
Oct 29 22:25:09.301674 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:25:09.301647    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda28494b_2ff0_4782_b53f_2583dc161b9d.slice/crio-20c0d007bc06d0921b37588a472d9dff97c760d5187bf6cde8b12ba19ac086f7 WatchSource:0}: Error finding container 20c0d007bc06d0921b37588a472d9dff97c760d5187bf6cde8b12ba19ac086f7: Status 404 returned error can't find the container with id 20c0d007bc06d0921b37588a472d9dff97c760d5187bf6cde8b12ba19ac086f7
Oct 29 22:25:09.822235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:09.822200    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:10.003004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:10.002967    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" event={"ID":"da28494b-2ff0-4782-b53f-2583dc161b9d","Type":"ContainerStarted","Data":"88c78d18f392f799fd57b5f221de27050544be75f39549dd0979b37702b90a17"}
Oct 29 22:25:10.003004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:10.003010    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" event={"ID":"da28494b-2ff0-4782-b53f-2583dc161b9d","Type":"ContainerStarted","Data":"20c0d007bc06d0921b37588a472d9dff97c760d5187bf6cde8b12ba19ac086f7"}
Oct 29 22:25:10.008473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:10.008445    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:10.046228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:10.046186    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podStartSLOduration=1.863152397 podStartE2EDuration="2.046172818s" totalImagesPullingTime="183.020421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:10.044537738 +0000 UTC m=+652.069313889" watchObservedRunningTime="2026-10-29 22:25:10.046172818 +0000 UTC m=+652.070948948"
Oct 29 22:25:11.514058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.514025    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"]
Oct 29 22:25:11.517881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.517859    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.547683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.547662    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 29 22:25:11.549864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.549845    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"]
Oct 29 22:25:11.588434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.588413    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:11.658255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.658231    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faba6185-f761-4f97-b100-653f1d78ca35-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.658399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.658260    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faba6185-f761-4f97-b100-653f1d78ca35-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.658399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.658302    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxndd\" (UniqueName: \"kubernetes.io/projected/faba6185-f761-4f97-b100-653f1d78ca35-kube-api-access-pxndd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.658487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.658443    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faba6185-f761-4f97-b100-653f1d78ca35-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.658487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.658471    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faba6185-f761-4f97-b100-653f1d78ca35-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.658572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.658486    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faba6185-f761-4f97-b100-653f1d78ca35-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.758962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.758937    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxndd\" (UniqueName: \"kubernetes.io/projected/faba6185-f761-4f97-b100-653f1d78ca35-kube-api-access-pxndd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.759066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.758966    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faba6185-f761-4f97-b100-653f1d78ca35-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.759066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.758992    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faba6185-f761-4f97-b100-653f1d78ca35-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.759066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.759013    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faba6185-f761-4f97-b100-653f1d78ca35-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.759066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.759063    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faba6185-f761-4f97-b100-653f1d78ca35-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.759266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.759095    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faba6185-f761-4f97-b100-653f1d78ca35-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.759510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.759486    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faba6185-f761-4f97-b100-653f1d78ca35-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.759604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.759529    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faba6185-f761-4f97-b100-653f1d78ca35-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.759604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.759553    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faba6185-f761-4f97-b100-653f1d78ca35-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.761240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.761220    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faba6185-f761-4f97-b100-653f1d78ca35-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.761550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.761534    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faba6185-f761-4f97-b100-653f1d78ca35-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.779033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.778974    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxndd\" (UniqueName: \"kubernetes.io/projected/faba6185-f761-4f97-b100-653f1d78ca35-kube-api-access-pxndd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs\" (UID: \"faba6185-f761-4f97-b100-653f1d78ca35\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:11.831598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:11.831577    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:12.125372 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:25:12.125317    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaba6185_f761_4f97_b100_653f1d78ca35.slice/crio-acf12d0e09a7f34b837e398565db5715cb7108740fcfade6672cdbd13293981b WatchSource:0}: Error finding container acf12d0e09a7f34b837e398565db5715cb7108740fcfade6672cdbd13293981b: Status 404 returned error can't find the container with id acf12d0e09a7f34b837e398565db5715cb7108740fcfade6672cdbd13293981b
Oct 29 22:25:12.126599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:12.126575    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"]
Oct 29 22:25:13.028690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:13.028649    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs" event={"ID":"faba6185-f761-4f97-b100-653f1d78ca35","Type":"ContainerStarted","Data":"50589d273a90cd2c6d63aff28bab86668a6f9a0733367441ff567e625a83dca1"}
Oct 29 22:25:13.029102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:13.028725    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs" event={"ID":"faba6185-f761-4f97-b100-653f1d78ca35","Type":"ContainerStarted","Data":"acf12d0e09a7f34b837e398565db5715cb7108740fcfade6672cdbd13293981b"}
Oct 29 22:25:13.071348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:13.071290    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs" podStartSLOduration=1.85971017 podStartE2EDuration="2.071276174s" totalImagesPullingTime="211.566004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:13.069725567 +0000 UTC m=+655.094501717" watchObservedRunningTime="2026-10-29 22:25:13.071276174 +0000 UTC m=+655.096052304"
Oct 29 22:25:13.832284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:13.832251    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:14.052941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:14.052912    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:15.184193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:15.184163    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:15.196554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:15.196532    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:16.061344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:16.060594    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2p75h"
Oct 29 22:25:16.380428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:16.380299    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:17.862826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:17.862792    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"]
Oct 29 22:25:17.866836 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:17.866807    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:17.899678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:17.899636    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 29 22:25:17.902870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:17.902835    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"]
Oct 29 22:25:17.957269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:17.957234    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:18.027802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.027764    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.027970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.027827    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.027970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.027859    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.027970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.027887    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cgqq\" (UniqueName: \"kubernetes.io/projected/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-kube-api-access-4cgqq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.028082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.028052    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.028128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.028110    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.055243 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:25:18.055208    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:25:18.128822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.128743    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.128822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.128795    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cgqq\" (UniqueName: \"kubernetes.io/projected/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-kube-api-access-4cgqq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.129057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.128845    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.129057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.128894    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.129057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.128962    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.129057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.129005    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.129269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.129137    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.129376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.129356    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.129473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.129452    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.131121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.131103    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.145158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.145136    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 29 22:25:18.151635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.151600    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.153739 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.153719    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cgqq\" (UniqueName: \"kubernetes.io/projected/82c8ba42-03e9-4192-a9af-6bdb5ceec1a5-kube-api-access-4cgqq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb\" (UID: \"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.181166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.181111    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:18.481598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:18.481571    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"]
Oct 29 22:25:18.481877 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:25:18.481856    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c8ba42_03e9_4192_a9af_6bdb5ceec1a5.slice/crio-dde5276c1e871107e102fe0e73620e24248b6d1d373c2762f573d30cce6e78c2 WatchSource:0}: Error finding container dde5276c1e871107e102fe0e73620e24248b6d1d373c2762f573d30cce6e78c2: Status 404 returned error can't find the container with id dde5276c1e871107e102fe0e73620e24248b6d1d373c2762f573d30cce6e78c2
Oct 29 22:25:19.009307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:19.009234    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:19.021065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:19.021045    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:19.063036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:19.062630    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb" event={"ID":"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5","Type":"ContainerStarted","Data":"7994db0f8b7c2b0b19abfc10611460fb4a11f22bb768f8c8c83a623d67e89991"}
Oct 29 22:25:19.063036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:19.062693    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb" event={"ID":"82c8ba42-03e9-4192-a9af-6bdb5ceec1a5","Type":"ContainerStarted","Data":"dde5276c1e871107e102fe0e73620e24248b6d1d373c2762f573d30cce6e78c2"}
Oct 29 22:25:19.073918 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:19.073895    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:25:19.105539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:19.105502    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb" podStartSLOduration=1.885368361 podStartE2EDuration="2.105488249s" totalImagesPullingTime="220.119888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:19.104156956 +0000 UTC m=+661.128933087" watchObservedRunningTime="2026-10-29 22:25:19.105488249 +0000 UTC m=+661.130264379"
Oct 29 22:25:19.181273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:19.181245    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:19.369048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:19.368976    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:21.832395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:21.832362    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:21.844657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:21.844628    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:22.089951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:22.089867    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs"
Oct 29 22:25:26.763611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.763573    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"]
Oct 29 22:25:26.767290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.767272    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.802587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.802559    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24145e7e-9d32-42a0-a462-dca182830c13-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.802694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.802606    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.802734 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.802702    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.802734 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.802730    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.802821 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.802746    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.802821 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.802766    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.802897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.802870    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24145e7e-9d32-42a0-a462-dca182830c13-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.802932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.802902    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rll8j\" (UniqueName: \"kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-kube-api-access-rll8j\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.802932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.802925    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.809237 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.809215    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 29 22:25:26.823422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.823401    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-67ntj\""
Oct 29 22:25:26.826524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.826504    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"]
Oct 29 22:25:26.829887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.829873    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"]
Oct 29 22:25:26.829965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.829955    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:26.860315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.860293    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"]
Oct 29 22:25:26.904179 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904146    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt2dn\" (UniqueName: \"kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-kube-api-access-mt2dn\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:26.904179 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904179    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:26.904388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904204    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.904388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904309    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.904388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904368    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.904514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904402    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4761326c-38e8-466e-8946-3c75605d0ad7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:26.904514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904431    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.904623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904511    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:26.904623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904562    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24145e7e-9d32-42a0-a462-dca182830c13-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.904623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904606    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rll8j\" (UniqueName: \"kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-kube-api-access-rll8j\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.904758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904665    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.904758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904674    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.904758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904671    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:26.904758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904723    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.904950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904768    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:26.904950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904809    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.904950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904851    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4761326c-38e8-466e-8946-3c75605d0ad7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:26.904950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.904942    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:26.905104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.905024    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:26.905104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.905098    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24145e7e-9d32-42a0-a462-dca182830c13-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.905178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.905151    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.905428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.905409    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.905866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.905844    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24145e7e-9d32-42a0-a462-dca182830c13-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.906780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.906760    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.907160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.907140    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24145e7e-9d32-42a0-a462-dca182830c13-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.925870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.925841    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:26.926411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:26.926342    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rll8j\" (UniqueName: \"kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-kube-api-access-rll8j\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-fhw57\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:27.005854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.005832    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.005932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.005862    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.005932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.005884    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4761326c-38e8-466e-8946-3c75605d0ad7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.006093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.006062    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.006186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.006117    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.006238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.006199    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt2dn\" (UniqueName: \"kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-kube-api-access-mt2dn\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.006291 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.006233    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.006291 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.006248    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.006422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.006297    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4761326c-38e8-466e-8946-3c75605d0ad7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.006422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.006362    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.006532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.006468    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.006596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.006549    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.006697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.006678    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.006873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.006855    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4761326c-38e8-466e-8946-3c75605d0ad7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.008163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.008147    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4761326c-38e8-466e-8946-3c75605d0ad7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.008529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.008509    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.026049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.025998    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.026225 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.026209    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt2dn\" (UniqueName: \"kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-kube-api-access-mt2dn\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-nqsf5\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.079985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.079967    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:27.145529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.145499    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:27.386107 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:25:27.386076    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24145e7e_9d32_42a0_a462_dca182830c13.slice/crio-0bf2dcf97e7ec66a5375c63f9d3a18179a32c260e4fe70d0e5188199d98f0bf6 WatchSource:0}: Error finding container 0bf2dcf97e7ec66a5375c63f9d3a18179a32c260e4fe70d0e5188199d98f0bf6: Status 404 returned error can't find the container with id 0bf2dcf97e7ec66a5375c63f9d3a18179a32c260e4fe70d0e5188199d98f0bf6
Oct 29 22:25:27.386651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.386631    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"]
Oct 29 22:25:27.449178 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:25:27.449154    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4761326c_38e8_466e_8946_3c75605d0ad7.slice/crio-fbe26575d84ace02e9c1c6950e15a9fabf0c803c8324b142d718f27f2ac752e5 WatchSource:0}: Error finding container fbe26575d84ace02e9c1c6950e15a9fabf0c803c8324b142d718f27f2ac752e5: Status 404 returned error can't find the container with id fbe26575d84ace02e9c1c6950e15a9fabf0c803c8324b142d718f27f2ac752e5
Oct 29 22:25:27.450516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:27.450491    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"]
Oct 29 22:25:28.130098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:28.130062    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57" event={"ID":"24145e7e-9d32-42a0-a462-dca182830c13","Type":"ContainerStarted","Data":"0bf2dcf97e7ec66a5375c63f9d3a18179a32c260e4fe70d0e5188199d98f0bf6"}
Oct 29 22:25:28.133416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:28.133356    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" event={"ID":"4761326c-38e8-466e-8946-3c75605d0ad7","Type":"ContainerStarted","Data":"fbe26575d84ace02e9c1c6950e15a9fabf0c803c8324b142d718f27f2ac752e5"}
Oct 29 22:25:28.182916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:28.182195    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:28.201430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:28.201402    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:29.153958 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:29.153915    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb"
Oct 29 22:25:30.325905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:30.325865    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:25:30.326288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:30.325949    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:25:30.326288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:30.325996    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:25:30.341457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:30.341425    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:25:30.341529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:30.341499    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:25:30.341571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:30.341529    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:25:31.153782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:31.153749    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57" event={"ID":"24145e7e-9d32-42a0-a462-dca182830c13","Type":"ContainerStarted","Data":"5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e"}
Oct 29 22:25:31.155660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:31.155636    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" event={"ID":"4761326c-38e8-466e-8946-3c75605d0ad7","Type":"ContainerStarted","Data":"956f3c1df683e571a0088062426798ed37c6fe83c31684139b60188912b5cfbb"}
Oct 29 22:25:31.244891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:31.244839    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57" podStartSLOduration=2.292304954 podStartE2EDuration="5.244821572s" totalImagesPullingTime="2.952516618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:31.230689241 +0000 UTC m=+673.255465372" watchObservedRunningTime="2026-10-29 22:25:31.244821572 +0000 UTC m=+673.269597702"
Oct 29 22:25:32.090350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:32.088681    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:32.093789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:32.093442    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-fhw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 29 22:25:32.093789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:32.093505    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57" podUID="24145e7e-9d32-42a0-a462-dca182830c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 29 22:25:32.145712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:32.145668    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:32.146938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:32.146906    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 29 22:25:32.147171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:32.147151    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" podUID="4761326c-38e8-466e-8946-3c75605d0ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 29 22:25:33.081224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:33.081183    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-fhw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 29 22:25:33.081615 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:33.081498    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57" podUID="24145e7e-9d32-42a0-a462-dca182830c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 29 22:25:33.145892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:33.145856    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 29 22:25:33.146401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:33.145924    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" podUID="4761326c-38e8-466e-8946-3c75605d0ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 29 22:25:34.088507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:34.088472    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-fhw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 29 22:25:34.088808 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:34.088778    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57" podUID="24145e7e-9d32-42a0-a462-dca182830c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 29 22:25:34.146791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:34.146739    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 29 22:25:34.147217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:34.146797    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" podUID="4761326c-38e8-466e-8946-3c75605d0ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 29 22:25:35.086350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.085187    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-fhw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 29 22:25:35.086350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.085254    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57" podUID="24145e7e-9d32-42a0-a462-dca182830c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 29 22:25:35.146936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.146603    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 29 22:25:35.146936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.146653    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" podUID="4761326c-38e8-466e-8946-3c75605d0ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 29 22:25:35.310350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.309600    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" podStartSLOduration=6.435630192 podStartE2EDuration="9.309580781s" totalImagesPullingTime="2.873950589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:31.291567492 +0000 UTC m=+673.316343623" watchObservedRunningTime="2026-10-29 22:25:35.309580781 +0000 UTC m=+677.334356941"
Oct 29 22:25:35.310350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.310156    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c78df8ff-z7jjx"]
Oct 29 22:25:35.320273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.320239    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c78df8ff-z7jjx"
Oct 29 22:25:35.330591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.330563    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c78df8ff-z7jjx"]
Oct 29 22:25:35.428853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.428760    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/876e4ae8-d438-444e-94fa-d603285f7fd8-tls-cert\") pod \"authorino-56c78df8ff-z7jjx\" (UID: \"876e4ae8-d438-444e-94fa-d603285f7fd8\") " pod="kuadrant-system/authorino-56c78df8ff-z7jjx"
Oct 29 22:25:35.428853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.428836    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmb7t\" (UniqueName: \"kubernetes.io/projected/876e4ae8-d438-444e-94fa-d603285f7fd8-kube-api-access-fmb7t\") pod \"authorino-56c78df8ff-z7jjx\" (UID: \"876e4ae8-d438-444e-94fa-d603285f7fd8\") " pod="kuadrant-system/authorino-56c78df8ff-z7jjx"
Oct 29 22:25:35.529867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.529831    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/876e4ae8-d438-444e-94fa-d603285f7fd8-tls-cert\") pod \"authorino-56c78df8ff-z7jjx\" (UID: \"876e4ae8-d438-444e-94fa-d603285f7fd8\") " pod="kuadrant-system/authorino-56c78df8ff-z7jjx"
Oct 29 22:25:35.530048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.529915    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmb7t\" (UniqueName: \"kubernetes.io/projected/876e4ae8-d438-444e-94fa-d603285f7fd8-kube-api-access-fmb7t\") pod \"authorino-56c78df8ff-z7jjx\" (UID: \"876e4ae8-d438-444e-94fa-d603285f7fd8\") " pod="kuadrant-system/authorino-56c78df8ff-z7jjx"
Oct 29 22:25:35.533230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.533202    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/876e4ae8-d438-444e-94fa-d603285f7fd8-tls-cert\") pod \"authorino-56c78df8ff-z7jjx\" (UID: \"876e4ae8-d438-444e-94fa-d603285f7fd8\") " pod="kuadrant-system/authorino-56c78df8ff-z7jjx"
Oct 29 22:25:35.550107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.550081    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmb7t\" (UniqueName: \"kubernetes.io/projected/876e4ae8-d438-444e-94fa-d603285f7fd8-kube-api-access-fmb7t\") pod \"authorino-56c78df8ff-z7jjx\" (UID: \"876e4ae8-d438-444e-94fa-d603285f7fd8\") " pod="kuadrant-system/authorino-56c78df8ff-z7jjx"
Oct 29 22:25:35.632395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.632355    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c78df8ff-z7jjx"
Oct 29 22:25:35.993672 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:35.992151    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c78df8ff-z7jjx"]
Oct 29 22:25:36.084117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:36.083676    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-fhw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 29 22:25:36.084117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:36.083733    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57" podUID="24145e7e-9d32-42a0-a462-dca182830c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 29 22:25:36.146209 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:36.146168    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 29 22:25:36.146424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:36.146244    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" podUID="4761326c-38e8-466e-8946-3c75605d0ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 29 22:25:36.229017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:36.228984    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c78df8ff-z7jjx" event={"ID":"876e4ae8-d438-444e-94fa-d603285f7fd8","Type":"ContainerStarted","Data":"1bbf133d12efe454e2d4f5cf39c5dd75dfa8dff11f28dc920402b89fd46ad7cd"}
Oct 29 22:25:37.084507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.084006    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-fhw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 29 22:25:37.084507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.084084    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57" podUID="24145e7e-9d32-42a0-a462-dca182830c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 29 22:25:37.147281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.147233    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 29 22:25:37.147633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.147582    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" podUID="4761326c-38e8-466e-8946-3c75605d0ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 29 22:25:37.239475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.239433    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c78df8ff-z7jjx" event={"ID":"876e4ae8-d438-444e-94fa-d603285f7fd8","Type":"ContainerStarted","Data":"bbffa002fa45ea9ce1e911f9a53fd335a4295017994ffdf819932f2911bfc29c"}
Oct 29 22:25:37.283994 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.283919    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c78df8ff-z7jjx" podStartSLOduration=1.721406257 podStartE2EDuration="2.283900063s" totalImagesPullingTime="562.493806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:37.28062232 +0000 UTC m=+679.305398488" watchObservedRunningTime="2026-10-29 22:25:37.283900063 +0000 UTC m=+679.308676198"
Oct 29 22:25:37.314392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.314356    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-955745748-t749f"]
Oct 29 22:25:37.315420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.314790    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-955745748-t749f" podUID="dbee6249-c27a-429b-836c-d9c1d9c86e6a" containerName="authorino" containerID="cri-o://a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09" gracePeriod=30
Oct 29 22:25:37.611648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.611569    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-955745748-t749f"
Oct 29 22:25:37.754479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.754435    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbee6249-c27a-429b-836c-d9c1d9c86e6a-kube-api-access-ks6gk\" (UniqueName: \"kubernetes.io/projected/dbee6249-c27a-429b-836c-d9c1d9c86e6a-kube-api-access-ks6gk\") pod \"dbee6249-c27a-429b-836c-d9c1d9c86e6a\" (UID: \"dbee6249-c27a-429b-836c-d9c1d9c86e6a\") "
Oct 29 22:25:37.754713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.754504    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbee6249-c27a-429b-836c-d9c1d9c86e6a-tls-cert\" (UniqueName: \"kubernetes.io/secret/dbee6249-c27a-429b-836c-d9c1d9c86e6a-tls-cert\") pod \"dbee6249-c27a-429b-836c-d9c1d9c86e6a\" (UID: \"dbee6249-c27a-429b-836c-d9c1d9c86e6a\") "
Oct 29 22:25:37.757334 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.757282    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbee6249-c27a-429b-836c-d9c1d9c86e6a-kube-api-access-ks6gk" pod "dbee6249-c27a-429b-836c-d9c1d9c86e6a" (UID: "dbee6249-c27a-429b-836c-d9c1d9c86e6a"). InnerVolumeSpecName "kube-api-access-ks6gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:25:37.770272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.770213    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbee6249-c27a-429b-836c-d9c1d9c86e6a-tls-cert" pod "dbee6249-c27a-429b-836c-d9c1d9c86e6a" (UID: "dbee6249-c27a-429b-836c-d9c1d9c86e6a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:25:37.855606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.855574    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks6gk\" (UniqueName: \"kubernetes.io/projected/dbee6249-c27a-429b-836c-d9c1d9c86e6a-kube-api-access-ks6gk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:25:37.855606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.855603    4860 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dbee6249-c27a-429b-836c-d9c1d9c86e6a-tls-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:25:37.873457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.873373    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:37.988671 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:37.988630    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:38.270106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:38.270066    4860 generic.go:356] "Generic (PLEG): container finished" podID="dbee6249-c27a-429b-836c-d9c1d9c86e6a" containerID="a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09" exitCode=0
Oct 29 22:25:38.271393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:38.271371    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-955745748-t749f"
Oct 29 22:25:38.272129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:38.271563    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-955745748-t749f" event={"ID":"dbee6249-c27a-429b-836c-d9c1d9c86e6a","Type":"ContainerDied","Data":"a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09"}
Oct 29 22:25:38.272288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:38.272267    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-955745748-t749f" event={"ID":"dbee6249-c27a-429b-836c-d9c1d9c86e6a","Type":"ContainerDied","Data":"d59b6beb2ba398475a5f7124bef3cca0328c8b995d0a1b0d9a198445c53660f6"}
Oct 29 22:25:38.272442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:38.272425    4860 scope.go:122] "RemoveContainer" containerID="a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09"
Oct 29 22:25:38.285818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:38.285800    4860 scope.go:122] "RemoveContainer" containerID="a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09"
Oct 29 22:25:38.286875 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:25:38.286250    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09\": container with ID starting with a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09 not found: ID does not exist" containerID="a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09"
Oct 29 22:25:38.286875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:38.286289    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09"} err="failed to get container status \"a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09\": rpc error: code = NotFound desc = could not find container \"a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09\": container with ID starting with a083d3a1ac3a25d51cdf54ac1482c1002591e07bee3e8b12477e9789d15baa09 not found: ID does not exist"
Oct 29 22:25:38.332431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:38.332384    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-955745748-t749f"]
Oct 29 22:25:38.339251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:38.339213    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-955745748-t749f"]
Oct 29 22:25:39.033662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:39.033624    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:39.034130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:39.034087    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:39.035257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:39.035242    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:25:39.147425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:39.147377    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:25:39.147622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:39.147454    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" podUID="4761326c-38e8-466e-8946-3c75605d0ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:25:39.260573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:39.259841    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:39.260573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:39.260543    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:39.263173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:39.261542    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:25:40.111694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:40.111658    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbee6249-c27a-429b-836c-d9c1d9c86e6a" path="/var/lib/kubelet/pods/dbee6249-c27a-429b-836c-d9c1d9c86e6a/volumes"
Oct 29 22:25:42.027914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.027878    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:42.123858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.123821    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-8wn8h"]
Oct 29 22:25:42.124252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.124234    4860 cpu_manager.go:414] "containerMap: removing container" podUID="dbee6249-c27a-429b-836c-d9c1d9c86e6a" containerName="authorino"
Oct 29 22:25:42.124372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.124254    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbee6249-c27a-429b-836c-d9c1d9c86e6a" containerName="authorino"
Oct 29 22:25:42.124372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.124367    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbee6249-c27a-429b-836c-d9c1d9c86e6a" containerName="authorino"
Oct 29 22:25:42.130357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.130316    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h"
Oct 29 22:25:42.145846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.145820    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-8wn8h"]
Oct 29 22:25:42.154004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.153980    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:42.157691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.157668    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 29 22:25:42.157798 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.157729    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9fzhv\""
Oct 29 22:25:42.161829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.161802    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"]
Oct 29 22:25:42.165774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.165757    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"
Oct 29 22:25:42.182694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.182672    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"]
Oct 29 22:25:42.310782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.310690    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5ce19f-bfe6-41bd-9c75-70dd54594ea3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-64t9f\" (UID: \"ac5ce19f-bfe6-41bd-9c75-70dd54594ea3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"
Oct 29 22:25:42.310782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.310758    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw26q\" (UniqueName: \"kubernetes.io/projected/23a26fe4-1010-464f-9df0-ff85860cc488-kube-api-access-tw26q\") pod \"payload-processing-5f78687d5b-8wn8h\" (UID: \"23a26fe4-1010-464f-9df0-ff85860cc488\") " pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h"
Oct 29 22:25:42.310999 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.310810    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a26fe4-1010-464f-9df0-ff85860cc488-plugins-config-volume\") pod \"payload-processing-5f78687d5b-8wn8h\" (UID: \"23a26fe4-1010-464f-9df0-ff85860cc488\") " pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h"
Oct 29 22:25:42.310999 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.310905    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqpmz\" (UniqueName: \"kubernetes.io/projected/ac5ce19f-bfe6-41bd-9c75-70dd54594ea3-kube-api-access-cqpmz\") pod \"payload-pre-processing-7b98bb678c-64t9f\" (UID: \"ac5ce19f-bfe6-41bd-9c75-70dd54594ea3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"
Oct 29 22:25:42.412465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.412421    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5ce19f-bfe6-41bd-9c75-70dd54594ea3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-64t9f\" (UID: \"ac5ce19f-bfe6-41bd-9c75-70dd54594ea3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"
Oct 29 22:25:42.412633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.412509    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw26q\" (UniqueName: \"kubernetes.io/projected/23a26fe4-1010-464f-9df0-ff85860cc488-kube-api-access-tw26q\") pod \"payload-processing-5f78687d5b-8wn8h\" (UID: \"23a26fe4-1010-464f-9df0-ff85860cc488\") " pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h"
Oct 29 22:25:42.412633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.412538    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a26fe4-1010-464f-9df0-ff85860cc488-plugins-config-volume\") pod \"payload-processing-5f78687d5b-8wn8h\" (UID: \"23a26fe4-1010-464f-9df0-ff85860cc488\") " pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h"
Oct 29 22:25:42.412633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.412627    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqpmz\" (UniqueName: \"kubernetes.io/projected/ac5ce19f-bfe6-41bd-9c75-70dd54594ea3-kube-api-access-cqpmz\") pod \"payload-pre-processing-7b98bb678c-64t9f\" (UID: \"ac5ce19f-bfe6-41bd-9c75-70dd54594ea3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"
Oct 29 22:25:42.413480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.413453    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a26fe4-1010-464f-9df0-ff85860cc488-plugins-config-volume\") pod \"payload-processing-5f78687d5b-8wn8h\" (UID: \"23a26fe4-1010-464f-9df0-ff85860cc488\") " pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h"
Oct 29 22:25:42.413594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.413458    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5ce19f-bfe6-41bd-9c75-70dd54594ea3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-64t9f\" (UID: \"ac5ce19f-bfe6-41bd-9c75-70dd54594ea3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"
Oct 29 22:25:42.437385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.434119    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw26q\" (UniqueName: \"kubernetes.io/projected/23a26fe4-1010-464f-9df0-ff85860cc488-kube-api-access-tw26q\") pod \"payload-processing-5f78687d5b-8wn8h\" (UID: \"23a26fe4-1010-464f-9df0-ff85860cc488\") " pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h"
Oct 29 22:25:42.437385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.434228    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqpmz\" (UniqueName: \"kubernetes.io/projected/ac5ce19f-bfe6-41bd-9c75-70dd54594ea3-kube-api-access-cqpmz\") pod \"payload-pre-processing-7b98bb678c-64t9f\" (UID: \"ac5ce19f-bfe6-41bd-9c75-70dd54594ea3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"
Oct 29 22:25:42.443542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.443513    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h"
Oct 29 22:25:42.480512 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.480472    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"
Oct 29 22:25:42.743762 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:25:42.743734    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a26fe4_1010_464f_9df0_ff85860cc488.slice/crio-7d926b9181d0b996af4ac17ab8bac2eb5db6d23b867d2dd3a13a7f3336681cf8 WatchSource:0}: Error finding container 7d926b9181d0b996af4ac17ab8bac2eb5db6d23b867d2dd3a13a7f3336681cf8: Status 404 returned error can't find the container with id 7d926b9181d0b996af4ac17ab8bac2eb5db6d23b867d2dd3a13a7f3336681cf8
Oct 29 22:25:42.745374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.745315    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-8wn8h"]
Oct 29 22:25:42.815978 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:25:42.815950    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5ce19f_bfe6_41bd_9c75_70dd54594ea3.slice/crio-879466ee0c3f4b6d1fc73747597fa5f7603be8ccb5ffb88e6697237b35d3d534 WatchSource:0}: Error finding container 879466ee0c3f4b6d1fc73747597fa5f7603be8ccb5ffb88e6697237b35d3d534: Status 404 returned error can't find the container with id 879466ee0c3f4b6d1fc73747597fa5f7603be8ccb5ffb88e6697237b35d3d534
Oct 29 22:25:42.818108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:42.818082    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"]
Oct 29 22:25:43.306669 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:43.306617    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h" event={"ID":"23a26fe4-1010-464f-9df0-ff85860cc488","Type":"ContainerStarted","Data":"7d926b9181d0b996af4ac17ab8bac2eb5db6d23b867d2dd3a13a7f3336681cf8"}
Oct 29 22:25:43.309021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:43.308986    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f" event={"ID":"ac5ce19f-bfe6-41bd-9c75-70dd54594ea3","Type":"ContainerStarted","Data":"879466ee0c3f4b6d1fc73747597fa5f7603be8ccb5ffb88e6697237b35d3d534"}
Oct 29 22:25:45.378653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:45.378618    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:25:46.333439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:46.333397    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h" event={"ID":"23a26fe4-1010-464f-9df0-ff85860cc488","Type":"ContainerStarted","Data":"52c5e4e8fbef6c1c9db95c86765d7bfe03dc1ac9b36c3e2a49047f4a1f58ca65"}
Oct 29 22:25:46.333600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:46.333467    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h"
Oct 29 22:25:46.334987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:46.334959    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f" event={"ID":"ac5ce19f-bfe6-41bd-9c75-70dd54594ea3","Type":"ContainerStarted","Data":"e7f9e3bde68129c3b397ba8d033474c81963c523e9da8cc63245f25f4d1a99a7"}
Oct 29 22:25:46.335127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:46.335011    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"
Oct 29 22:25:46.375315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:46.375272    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h" podStartSLOduration=1.194861491 podStartE2EDuration="4.375259883s" totalImagesPullingTime="3.180398392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:46.374588718 +0000 UTC m=+688.399364868" watchObservedRunningTime="2026-10-29 22:25:46.375259883 +0000 UTC m=+688.400036014"
Oct 29 22:25:46.415141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:46.415101    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f" podStartSLOduration=1.373807946 podStartE2EDuration="4.415092834s" totalImagesPullingTime="3.041284888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:46.414226159 +0000 UTC m=+688.439002314" watchObservedRunningTime="2026-10-29 22:25:46.415092834 +0000 UTC m=+688.439868965"
Oct 29 22:25:48.055245 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:25:48.055210    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:25:52.411166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:52.411132    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5955b644bc-t5bnn"]
Oct 29 22:25:52.415945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:52.415925    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn"
Oct 29 22:25:52.432944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:52.432921    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5955b644bc-t5bnn"]
Oct 29 22:25:52.444271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:52.444248    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 29 22:25:52.444271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:52.444256    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 29 22:25:52.445685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:52.445669    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-2f5bf\""
Oct 29 22:25:52.611024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:52.610999    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6vlx\" (UniqueName: \"kubernetes.io/projected/a5df48c6-245a-4387-8d8f-745e0ef5d0f1-kube-api-access-h6vlx\") pod \"ai-gateway-controller-5955b644bc-t5bnn\" (UID: \"a5df48c6-245a-4387-8d8f-745e0ef5d0f1\") " pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn"
Oct 29 22:25:52.611140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:52.611062    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5df48c6-245a-4387-8d8f-745e0ef5d0f1-metrics-certs\") pod \"ai-gateway-controller-5955b644bc-t5bnn\" (UID: \"a5df48c6-245a-4387-8d8f-745e0ef5d0f1\") " pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn"
Oct 29 22:25:52.712401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:52.712371    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5df48c6-245a-4387-8d8f-745e0ef5d0f1-metrics-certs\") pod \"ai-gateway-controller-5955b644bc-t5bnn\" (UID: \"a5df48c6-245a-4387-8d8f-745e0ef5d0f1\") " pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn"
Oct 29 22:25:52.712526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:52.712441    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6vlx\" (UniqueName: \"kubernetes.io/projected/a5df48c6-245a-4387-8d8f-745e0ef5d0f1-kube-api-access-h6vlx\") pod \"ai-gateway-controller-5955b644bc-t5bnn\" (UID: \"a5df48c6-245a-4387-8d8f-745e0ef5d0f1\") " pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn"
Oct 29 22:25:52.714884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:52.714865    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5df48c6-245a-4387-8d8f-745e0ef5d0f1-metrics-certs\") pod \"ai-gateway-controller-5955b644bc-t5bnn\" (UID: \"a5df48c6-245a-4387-8d8f-745e0ef5d0f1\") " pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn"
Oct 29 22:25:52.733208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:52.733184    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6vlx\" (UniqueName: \"kubernetes.io/projected/a5df48c6-245a-4387-8d8f-745e0ef5d0f1-kube-api-access-h6vlx\") pod \"ai-gateway-controller-5955b644bc-t5bnn\" (UID: \"a5df48c6-245a-4387-8d8f-745e0ef5d0f1\") " pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn"
Oct 29 22:25:53.027987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:53.027904    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn"
Oct 29 22:25:53.323498 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:25:53.323471    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5df48c6_245a_4387_8d8f_745e0ef5d0f1.slice/crio-b67c0f2fbd16ad59b55e23176ff4e2c524c41e41a4aee80405b343a4a7911674 WatchSource:0}: Error finding container b67c0f2fbd16ad59b55e23176ff4e2c524c41e41a4aee80405b343a4a7911674: Status 404 returned error can't find the container with id b67c0f2fbd16ad59b55e23176ff4e2c524c41e41a4aee80405b343a4a7911674
Oct 29 22:25:53.323777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:53.323754    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5955b644bc-t5bnn"]
Oct 29 22:25:53.370655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:53.370631    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn" event={"ID":"a5df48c6-245a-4387-8d8f-745e0ef5d0f1","Type":"ContainerStarted","Data":"b67c0f2fbd16ad59b55e23176ff4e2c524c41e41a4aee80405b343a4a7911674"}
Oct 29 22:25:56.387424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:56.387378    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn" event={"ID":"a5df48c6-245a-4387-8d8f-745e0ef5d0f1","Type":"ContainerStarted","Data":"b04014586950a8ed680bd12fef69a7f41c07ac3921c513ffa4939749d0617979"}
Oct 29 22:25:56.387831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:56.387480    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn"
Oct 29 22:25:56.433024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:56.432977    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn" podStartSLOduration=2.278725736 podStartE2EDuration="4.432964331s" totalImagesPullingTime="2.154238595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:56.430839039 +0000 UTC m=+698.455615183" watchObservedRunningTime="2026-10-29 22:25:56.432964331 +0000 UTC m=+698.457740461"
Oct 29 22:25:57.343198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:57.343172    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-8wn8h"
Oct 29 22:25:57.343393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:25:57.343343    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-64t9f"
Oct 29 22:26:07.262195 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:07.262162    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:07.378917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:07.378888    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:07.395744 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:07.395723    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5955b644bc-t5bnn"
Oct 29 22:26:09.134601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:09.134572    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:09.657609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:09.657580    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:09.943677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:09.943650    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:11.230410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:11.230362    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:11.417007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:11.416979    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:12.699601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:12.699569    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:12.816040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:12.816012    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:18.058996 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:26:18.058955    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:26:27.320439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:27.320407    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:34.079385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.079346    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:34.198970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.198937    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:34.527472 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.527437    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"]
Oct 29 22:26:34.532130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.532105    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.550517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.550493    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-784e7a-openshift-default-dockercfg-hn6qv\""
Oct 29 22:26:34.566565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.566539    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"]
Oct 29 22:26:34.640317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.640293    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.640462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.640350    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.640462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.640414    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc22e15-7269-4307-a9bf-ad39641f7287-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.640548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.640491    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.640548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.640528    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.640610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.640547    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d45t\" (UniqueName: \"kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-kube-api-access-4d45t\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.640610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.640586    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.640676 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.640620    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.640708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.640678    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.701860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.701834    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:34.741614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.741589    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.741738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.741638    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.741853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.741835    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d45t\" (UniqueName: \"kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-kube-api-access-4d45t\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.741915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.741901    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.741971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.741959    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.742096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.742076    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.742299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.742275    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.742299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.742295    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.742470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.742316    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.742470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.742107    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.742470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.742382    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.742470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.742429    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc22e15-7269-4307-a9bf-ad39641f7287-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.742673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.742627    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.743122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.743099    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc22e15-7269-4307-a9bf-ad39641f7287-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.744413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.744390    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.744413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.744407    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.767499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.767473    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d45t\" (UniqueName: \"kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-kube-api-access-4d45t\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.768272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.768249    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.845461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.845402    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:34.858696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.858666    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:34.958184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.958156    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"]
Oct 29 22:26:34.962011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.961993    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:34.979889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.979862    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fc684b-openshift-default-dockercfg-8hgq7\""
Oct 29 22:26:34.985187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.985164    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"]
Oct 29 22:26:34.995049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:34.995004    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:35.046208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.046144    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.046313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.046253    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da28397d-fefc-4a03-8dc8-3fad87558566-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.046313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.046280    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4h6t\" (UniqueName: \"kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-kube-api-access-b4h6t\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.046471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.046311    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.046521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.046467    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.046521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.046506    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.046632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.046614    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.046726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.046703    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da28397d-fefc-4a03-8dc8-3fad87558566-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.046826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.046804    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.136177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.136110    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:35.147835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.147810    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da28397d-fefc-4a03-8dc8-3fad87558566-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.147947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.147846    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.147947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.147876    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.147947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.147915    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da28397d-fefc-4a03-8dc8-3fad87558566-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.147947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.147933    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4h6t\" (UniqueName: \"kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-kube-api-access-b4h6t\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.148193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.147953    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.148193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.147996    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.148193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.148027    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.148193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.148045    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.148424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.148374    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.148817 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.148792    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da28397d-fefc-4a03-8dc8-3fad87558566-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.148971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.148953    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.149032 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.148990    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.149032 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.149019    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.150896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.150870    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.151008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.150993    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da28397d-fefc-4a03-8dc8-3fad87558566-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.160444 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:26:35.160420    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc22e15_7269_4307_a9bf_ad39641f7287.slice/crio-4bfa3eb67207460a9119517981670c7d5b8a4198a881ca98166417c024fc904b WatchSource:0}: Error finding container 4bfa3eb67207460a9119517981670c7d5b8a4198a881ca98166417c024fc904b: Status 404 returned error can't find the container with id 4bfa3eb67207460a9119517981670c7d5b8a4198a881ca98166417c024fc904b
Oct 29 22:26:35.161236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.161219    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"]
Oct 29 22:26:35.162506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.162479    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:26:35.162573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.162537    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:26:35.162613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.162571    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:26:35.170023 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.170004    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.170334 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.170297    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4h6t\" (UniqueName: \"kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-kube-api-access-b4h6t\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.219686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.219661    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"]
Oct 29 22:26:35.224274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.224256    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.246340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.246287    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-08ebee-openshift-default-dockercfg-4sgmc\""
Oct 29 22:26:35.249881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.249857    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"]
Oct 29 22:26:35.268198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.268164    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:35.276563 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.276539    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:35.353533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.349559    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.353533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.349602    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.353533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.349625    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.353533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.349657    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.353533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.349676    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.353533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.349727    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.353533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.349760    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.353533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.349786    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.353533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.349813    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-489hs\" (UniqueName: \"kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-kube-api-access-489hs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.398975 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.398911    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:35.451316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.451278    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.451490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.451359    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-489hs\" (UniqueName: \"kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-kube-api-access-489hs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.451490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.451404    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.451490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.451449    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.451490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.451482    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.451711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.451531    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.451711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.451559    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.451711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.451642    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.451876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.451706    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.451876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.451854    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.452141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.452116    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.452235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.452178    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.452235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.452215    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.452343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.452250    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.454143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.454126    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.454414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.454397    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.472223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.472199    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-489hs\" (UniqueName: \"kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-kube-api-access-489hs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.472319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.472302    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.525685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.525664    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:35.538708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.538687    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:35.596202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.596172    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"]
Oct 29 22:26:35.601353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.601218    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" event={"ID":"bbc22e15-7269-4307-a9bf-ad39641f7287","Type":"ContainerStarted","Data":"53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c"}
Oct 29 22:26:35.601353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.601274    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" event={"ID":"bbc22e15-7269-4307-a9bf-ad39641f7287","Type":"ContainerStarted","Data":"4bfa3eb67207460a9119517981670c7d5b8a4198a881ca98166417c024fc904b"}
Oct 29 22:26:35.609104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.609063    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:26:35.609220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.609153    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:26:35.609220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.609198    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:26:35.655121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.655033    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" podStartSLOduration=1.6550130140000001 podStartE2EDuration="1.655013014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:35.651964181 +0000 UTC m=+737.676740364" watchObservedRunningTime="2026-10-29 22:26:35.655013014 +0000 UTC m=+737.679789148"
Oct 29 22:26:35.675475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.675451    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:35.845970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.845935    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:35.855868 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:26:35.855835    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec041e3f_8c83_4807_94f2_40a0636a4dd9.slice/crio-e181cb13b63897dfbe1131ce26d6fb5a8f37b9876a77c6c0f822d98caaf70e6b WatchSource:0}: Error finding container e181cb13b63897dfbe1131ce26d6fb5a8f37b9876a77c6c0f822d98caaf70e6b: Status 404 returned error can't find the container with id e181cb13b63897dfbe1131ce26d6fb5a8f37b9876a77c6c0f822d98caaf70e6b
Oct 29 22:26:35.856872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.856843    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"]
Oct 29 22:26:35.858565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.858528    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:26:35.858656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.858593    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:26:35.858656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:35.858622    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:26:36.607683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:36.607646    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" event={"ID":"da28397d-fefc-4a03-8dc8-3fad87558566","Type":"ContainerStarted","Data":"ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61"}
Oct 29 22:26:36.608148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:36.607709    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" event={"ID":"da28397d-fefc-4a03-8dc8-3fad87558566","Type":"ContainerStarted","Data":"d6c5c65696e047cc92ee428c15a77b25e1650f9166c35b9a61c3574853c42a6b"}
Oct 29 22:26:36.609542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:36.609512    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" event={"ID":"ec041e3f-8c83-4807-94f2-40a0636a4dd9","Type":"ContainerStarted","Data":"59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00"}
Oct 29 22:26:36.609651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:36.609563    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" event={"ID":"ec041e3f-8c83-4807-94f2-40a0636a4dd9","Type":"ContainerStarted","Data":"e181cb13b63897dfbe1131ce26d6fb5a8f37b9876a77c6c0f822d98caaf70e6b"}
Oct 29 22:26:36.655935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:36.655893    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" podStartSLOduration=2.655880197 podStartE2EDuration="2.655880197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:36.655767146 +0000 UTC m=+738.680543278" watchObservedRunningTime="2026-10-29 22:26:36.655880197 +0000 UTC m=+738.680656330"
Oct 29 22:26:36.700901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:36.700861    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" podStartSLOduration=1.7008500180000001 podStartE2EDuration="1.700850018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:36.698595517 +0000 UTC m=+738.723371647" watchObservedRunningTime="2026-10-29 22:26:36.700850018 +0000 UTC m=+738.725626149"
Oct 29 22:26:36.851107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:36.851077    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:37.276918 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:37.276891    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:37.281415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:37.281393    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:37.539337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:37.539247    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:37.544178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:37.544152    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:37.614596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:37.614562    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:37.615041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:37.614615    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:37.615041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:37.614641    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:37.615292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:37.615272    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:37.615413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:37.615365    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:26:37.615477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:37.615439    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:26:38.218873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:38.218830    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:38.333882 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:38.333848    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:39.108367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:39.108302    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:39.249663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:39.249633    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:39.393714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:39.393638    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:39.531094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:39.531041    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:39.665251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:39.665186    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:40.306314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.306281    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:40.497290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.497255    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:40.578312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.578238    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"]
Oct 29 22:26:40.582292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.582276    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"
Oct 29 22:26:40.605130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.605108    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"]
Oct 29 22:26:40.616157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.616139    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-784e7a\""
Oct 29 22:26:40.616335 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.616303    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-784e7a-dockercfg-g7x6r\""
Oct 29 22:26:40.659615 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.659589    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"]
Oct 29 22:26:40.663048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.663033    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"
Oct 29 22:26:40.685735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.685709    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"]
Oct 29 22:26:40.692437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.692415    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:40.706128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.706094    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaed535-f4bd-430d-8528-af2feb5a8ca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw\" (UID: \"4aaed535-f4bd-430d-8528-af2feb5a8ca4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"
Oct 29 22:26:40.706381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.706312    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnb86\" (UniqueName: \"kubernetes.io/projected/4aaed535-f4bd-430d-8528-af2feb5a8ca4-kube-api-access-hnb86\") pod \"payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw\" (UID: \"4aaed535-f4bd-430d-8528-af2feb5a8ca4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"
Oct 29 22:26:40.807785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.807749    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaed535-f4bd-430d-8528-af2feb5a8ca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw\" (UID: \"4aaed535-f4bd-430d-8528-af2feb5a8ca4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"
Oct 29 22:26:40.807945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.807803    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e951ea-8eb5-4a3a-aabf-dde45406c921-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx\" (UID: \"f5e951ea-8eb5-4a3a-aabf-dde45406c921\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"
Oct 29 22:26:40.807945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.807868    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnb86\" (UniqueName: \"kubernetes.io/projected/4aaed535-f4bd-430d-8528-af2feb5a8ca4-kube-api-access-hnb86\") pod \"payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw\" (UID: \"4aaed535-f4bd-430d-8528-af2feb5a8ca4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"
Oct 29 22:26:40.807945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.807902    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xg48\" (UniqueName: \"kubernetes.io/projected/f5e951ea-8eb5-4a3a-aabf-dde45406c921-kube-api-access-6xg48\") pod \"payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx\" (UID: \"f5e951ea-8eb5-4a3a-aabf-dde45406c921\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"
Oct 29 22:26:40.808464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.808445    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaed535-f4bd-430d-8528-af2feb5a8ca4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw\" (UID: \"4aaed535-f4bd-430d-8528-af2feb5a8ca4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"
Oct 29 22:26:40.829226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.829165    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnb86\" (UniqueName: \"kubernetes.io/projected/4aaed535-f4bd-430d-8528-af2feb5a8ca4-kube-api-access-hnb86\") pod \"payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw\" (UID: \"4aaed535-f4bd-430d-8528-af2feb5a8ca4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"
Oct 29 22:26:40.837854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.837826    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:40.896054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.896030    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"
Oct 29 22:26:40.909406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.909387    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xg48\" (UniqueName: \"kubernetes.io/projected/f5e951ea-8eb5-4a3a-aabf-dde45406c921-kube-api-access-6xg48\") pod \"payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx\" (UID: \"f5e951ea-8eb5-4a3a-aabf-dde45406c921\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"
Oct 29 22:26:40.909482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.909434    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e951ea-8eb5-4a3a-aabf-dde45406c921-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx\" (UID: \"f5e951ea-8eb5-4a3a-aabf-dde45406c921\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"
Oct 29 22:26:40.910070 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.910053    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e951ea-8eb5-4a3a-aabf-dde45406c921-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx\" (UID: \"f5e951ea-8eb5-4a3a-aabf-dde45406c921\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"
Oct 29 22:26:40.931297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.931277    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xg48\" (UniqueName: \"kubernetes.io/projected/f5e951ea-8eb5-4a3a-aabf-dde45406c921-kube-api-access-6xg48\") pod \"payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx\" (UID: \"f5e951ea-8eb5-4a3a-aabf-dde45406c921\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"
Oct 29 22:26:40.975707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:40.975681    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"
Oct 29 22:26:41.202487 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:26:41.202457    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aaed535_f4bd_430d_8528_af2feb5a8ca4.slice/crio-9ba8aef88994da67701631c165c10366b293d92bca55d9079a7edf14cd2c4e1a WatchSource:0}: Error finding container 9ba8aef88994da67701631c165c10366b293d92bca55d9079a7edf14cd2c4e1a: Status 404 returned error can't find the container with id 9ba8aef88994da67701631c165c10366b293d92bca55d9079a7edf14cd2c4e1a
Oct 29 22:26:41.203684 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:41.203657    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"]
Oct 29 22:26:41.275974 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:26:41.275950    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e951ea_8eb5_4a3a_aabf_dde45406c921.slice/crio-e40028696e1e6de26bc77ad1e7ab727c4c3e2250d5ffb2df5045fb379100881b WatchSource:0}: Error finding container e40028696e1e6de26bc77ad1e7ab727c4c3e2250d5ffb2df5045fb379100881b: Status 404 returned error can't find the container with id e40028696e1e6de26bc77ad1e7ab727c4c3e2250d5ffb2df5045fb379100881b
Oct 29 22:26:41.277531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:41.277508    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"]
Oct 29 22:26:41.638577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:41.638485    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw" event={"ID":"4aaed535-f4bd-430d-8528-af2feb5a8ca4","Type":"ContainerStarted","Data":"9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc"}
Oct 29 22:26:41.638995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:41.638648    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"
Oct 29 22:26:41.638995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:41.638679    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw" event={"ID":"4aaed535-f4bd-430d-8528-af2feb5a8ca4","Type":"ContainerStarted","Data":"9ba8aef88994da67701631c165c10366b293d92bca55d9079a7edf14cd2c4e1a"}
Oct 29 22:26:41.640184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:41.640159    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx" event={"ID":"f5e951ea-8eb5-4a3a-aabf-dde45406c921","Type":"ContainerStarted","Data":"c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0"}
Oct 29 22:26:41.640275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:41.640203    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx" event={"ID":"f5e951ea-8eb5-4a3a-aabf-dde45406c921","Type":"ContainerStarted","Data":"e40028696e1e6de26bc77ad1e7ab727c4c3e2250d5ffb2df5045fb379100881b"}
Oct 29 22:26:41.640275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:41.640240    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"
Oct 29 22:26:41.683579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:41.683542    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw" podStartSLOduration=1.6835288 podStartE2EDuration="1.6835288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:41.682200977 +0000 UTC m=+743.706977102" watchObservedRunningTime="2026-10-29 22:26:41.6835288 +0000 UTC m=+743.708304930"
Oct 29 22:26:41.725874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:41.725832    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx" podStartSLOduration=1.725820139 podStartE2EDuration="1.725820139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:41.724318862 +0000 UTC m=+743.749095004" watchObservedRunningTime="2026-10-29 22:26:41.725820139 +0000 UTC m=+743.750596296"
Oct 29 22:26:41.933062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:41.933023    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:42.055767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.055736    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:42.341213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.341133    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:42.487874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.487842    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"]
Oct 29 22:26:42.491507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.491491    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"
Oct 29 22:26:42.501649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.501622    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:42.510478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.510454    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-43b37d-dockercfg-rrdc7\""
Oct 29 22:26:42.511643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.511624    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"]
Oct 29 22:26:42.522688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.522663    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-43b37d\""
Oct 29 22:26:42.571368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.568332    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"]
Oct 29 22:26:42.572184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.572079    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"
Oct 29 22:26:42.589641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.589611    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"]
Oct 29 22:26:42.625930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.625860    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghsc\" (UniqueName: \"kubernetes.io/projected/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-kube-api-access-bghsc\") pod \"payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw\" (UID: \"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"
Oct 29 22:26:42.626074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.626006    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw\" (UID: \"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"
Oct 29 22:26:42.726898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.726864    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw\" (UID: \"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"
Oct 29 22:26:42.727256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.726921    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghsc\" (UniqueName: \"kubernetes.io/projected/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-kube-api-access-bghsc\") pod \"payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw\" (UID: \"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"
Oct 29 22:26:42.727256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.726971    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4md8k\" (UniqueName: \"kubernetes.io/projected/056308d8-3cce-4df1-a0a9-e0486218e4f5-kube-api-access-4md8k\") pod \"payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc\" (UID: \"056308d8-3cce-4df1-a0a9-e0486218e4f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"
Oct 29 22:26:42.727256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.727013    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056308d8-3cce-4df1-a0a9-e0486218e4f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc\" (UID: \"056308d8-3cce-4df1-a0a9-e0486218e4f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"
Oct 29 22:26:42.727701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.727682    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw\" (UID: \"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"
Oct 29 22:26:42.747566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.747539    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghsc\" (UniqueName: \"kubernetes.io/projected/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-kube-api-access-bghsc\") pod \"payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw\" (UID: \"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"
Oct 29 22:26:42.805393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.805365    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"
Oct 29 22:26:42.827807 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.827775    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056308d8-3cce-4df1-a0a9-e0486218e4f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc\" (UID: \"056308d8-3cce-4df1-a0a9-e0486218e4f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"
Oct 29 22:26:42.827983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.827952    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4md8k\" (UniqueName: \"kubernetes.io/projected/056308d8-3cce-4df1-a0a9-e0486218e4f5-kube-api-access-4md8k\") pod \"payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc\" (UID: \"056308d8-3cce-4df1-a0a9-e0486218e4f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"
Oct 29 22:26:42.828734 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.828709    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056308d8-3cce-4df1-a0a9-e0486218e4f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc\" (UID: \"056308d8-3cce-4df1-a0a9-e0486218e4f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"
Oct 29 22:26:42.848404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.848363    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4md8k\" (UniqueName: \"kubernetes.io/projected/056308d8-3cce-4df1-a0a9-e0486218e4f5-kube-api-access-4md8k\") pod \"payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc\" (UID: \"056308d8-3cce-4df1-a0a9-e0486218e4f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"
Oct 29 22:26:42.910878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:42.910812    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"
Oct 29 22:26:43.101612 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:26:43.101578    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a131e4_4792_4ab4_98d8_b4cdd6d7ad43.slice/crio-7f373f49aca1f17aca1ea88600e33cab5c58e9f84587b2477b4bb0d8e763944a WatchSource:0}: Error finding container 7f373f49aca1f17aca1ea88600e33cab5c58e9f84587b2477b4bb0d8e763944a: Status 404 returned error can't find the container with id 7f373f49aca1f17aca1ea88600e33cab5c58e9f84587b2477b4bb0d8e763944a
Oct 29 22:26:43.102797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:43.102772    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"]
Oct 29 22:26:43.209250 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:26:43.209226    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056308d8_3cce_4df1_a0a9_e0486218e4f5.slice/crio-b2bbda9a0050daa85c45c9a1cc698121383d621f62d1c6150455540e029de695 WatchSource:0}: Error finding container b2bbda9a0050daa85c45c9a1cc698121383d621f62d1c6150455540e029de695: Status 404 returned error can't find the container with id b2bbda9a0050daa85c45c9a1cc698121383d621f62d1c6150455540e029de695
Oct 29 22:26:43.210263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:43.210237    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"]
Oct 29 22:26:43.654109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:43.653987    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc" event={"ID":"056308d8-3cce-4df1-a0a9-e0486218e4f5","Type":"ContainerStarted","Data":"dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa"}
Oct 29 22:26:43.654109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:43.654053    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc" event={"ID":"056308d8-3cce-4df1-a0a9-e0486218e4f5","Type":"ContainerStarted","Data":"b2bbda9a0050daa85c45c9a1cc698121383d621f62d1c6150455540e029de695"}
Oct 29 22:26:43.654520 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:43.654252    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"
Oct 29 22:26:43.656615 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:43.656589    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw" event={"ID":"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43","Type":"ContainerStarted","Data":"f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0"}
Oct 29 22:26:43.656747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:43.656633    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw" event={"ID":"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43","Type":"ContainerStarted","Data":"7f373f49aca1f17aca1ea88600e33cab5c58e9f84587b2477b4bb0d8e763944a"}
Oct 29 22:26:43.656910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:43.656825    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"
Oct 29 22:26:43.695193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:43.695148    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc" podStartSLOduration=1.695131086 podStartE2EDuration="1.695131086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:43.693839623 +0000 UTC m=+745.718615753" watchObservedRunningTime="2026-10-29 22:26:43.695131086 +0000 UTC m=+745.719907217"
Oct 29 22:26:43.737831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:43.737786    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw" podStartSLOduration=1.737773856 podStartE2EDuration="1.737773856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:43.735068109 +0000 UTC m=+745.759844240" watchObservedRunningTime="2026-10-29 22:26:43.737773856 +0000 UTC m=+745.762549986"
Oct 29 22:26:43.993202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:43.993169    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:44.759657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:44.759621    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:44.853411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:44.853370    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"]
Oct 29 22:26:44.857081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:44.857061    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"
Oct 29 22:26:44.885931 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:44.885903    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:44.889943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:44.889922    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-2ba10a\""
Oct 29 22:26:44.892429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:44.892411    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"]
Oct 29 22:26:44.947582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:44.947557    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4fp8\" (UniqueName: \"kubernetes.io/projected/5962c518-2366-4739-995f-0caa8149b9be-kube-api-access-z4fp8\") pod \"payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv\" (UID: \"5962c518-2366-4739-995f-0caa8149b9be\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"
Oct 29 22:26:44.947689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:44.947614    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5962c518-2366-4739-995f-0caa8149b9be-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv\" (UID: \"5962c518-2366-4739-995f-0caa8149b9be\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"
Oct 29 22:26:45.049036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.048960    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4fp8\" (UniqueName: \"kubernetes.io/projected/5962c518-2366-4739-995f-0caa8149b9be-kube-api-access-z4fp8\") pod \"payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv\" (UID: \"5962c518-2366-4739-995f-0caa8149b9be\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"
Oct 29 22:26:45.049181 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.049042    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5962c518-2366-4739-995f-0caa8149b9be-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv\" (UID: \"5962c518-2366-4739-995f-0caa8149b9be\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"
Oct 29 22:26:45.049800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.049780    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5962c518-2366-4739-995f-0caa8149b9be-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv\" (UID: \"5962c518-2366-4739-995f-0caa8149b9be\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"
Oct 29 22:26:45.076274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.076251    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4fp8\" (UniqueName: \"kubernetes.io/projected/5962c518-2366-4739-995f-0caa8149b9be-kube-api-access-z4fp8\") pod \"payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv\" (UID: \"5962c518-2366-4739-995f-0caa8149b9be\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"
Oct 29 22:26:45.170411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.170382    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"
Oct 29 22:26:45.468352 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:26:45.468302    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5962c518_2366_4739_995f_0caa8149b9be.slice/crio-743398570cfa876de1d19a760105460e0561822afe4f2592d34d0bd455dc1f0d WatchSource:0}: Error finding container 743398570cfa876de1d19a760105460e0561822afe4f2592d34d0bd455dc1f0d: Status 404 returned error can't find the container with id 743398570cfa876de1d19a760105460e0561822afe4f2592d34d0bd455dc1f0d
Oct 29 22:26:45.468873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.468845    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"]
Oct 29 22:26:45.635558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.635506    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:45.678698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.678654    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv" event={"ID":"5962c518-2366-4739-995f-0caa8149b9be","Type":"ContainerStarted","Data":"bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2"}
Oct 29 22:26:45.678866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.678758    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv" event={"ID":"5962c518-2366-4739-995f-0caa8149b9be","Type":"ContainerStarted","Data":"743398570cfa876de1d19a760105460e0561822afe4f2592d34d0bd455dc1f0d"}
Oct 29 22:26:45.678866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.678792    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"
Oct 29 22:26:45.723498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.723408    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv" podStartSLOduration=1.7233962329999999 podStartE2EDuration="1.723396233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:45.721480805 +0000 UTC m=+747.746256957" watchObservedRunningTime="2026-10-29 22:26:45.723396233 +0000 UTC m=+747.748172355"
Oct 29 22:26:45.751897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.751852    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:45.915161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.906257    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"]
Oct 29 22:26:45.915161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.911738    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"
Oct 29 22:26:45.915716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.915559    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"]
Oct 29 22:26:45.934175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.934151    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-08ebee\""
Oct 29 22:26:45.957246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.954753    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:46.019440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.995746    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100f773a-93c3-49ef-b704-393f72d7b181-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q\" (UID: \"100f773a-93c3-49ef-b704-393f72d7b181\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"
Oct 29 22:26:46.019440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:45.995859    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvbnq\" (UniqueName: \"kubernetes.io/projected/100f773a-93c3-49ef-b704-393f72d7b181-kube-api-access-tvbnq\") pod \"payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q\" (UID: \"100f773a-93c3-49ef-b704-393f72d7b181\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"
Oct 29 22:26:46.029174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.029141    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-08ebee-dockercfg-6tv86\""
Oct 29 22:26:46.088712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.088671    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:46.096642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.096614    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100f773a-93c3-49ef-b704-393f72d7b181-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q\" (UID: \"100f773a-93c3-49ef-b704-393f72d7b181\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"
Oct 29 22:26:46.096781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.096645    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbnq\" (UniqueName: \"kubernetes.io/projected/100f773a-93c3-49ef-b704-393f72d7b181-kube-api-access-tvbnq\") pod \"payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q\" (UID: \"100f773a-93c3-49ef-b704-393f72d7b181\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"
Oct 29 22:26:46.097300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.097280    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100f773a-93c3-49ef-b704-393f72d7b181-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q\" (UID: \"100f773a-93c3-49ef-b704-393f72d7b181\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"
Oct 29 22:26:46.117702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.117671    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvbnq\" (UniqueName: \"kubernetes.io/projected/100f773a-93c3-49ef-b704-393f72d7b181-kube-api-access-tvbnq\") pod \"payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q\" (UID: \"100f773a-93c3-49ef-b704-393f72d7b181\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"
Oct 29 22:26:46.203785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.203759    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:46.230827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.230806    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"
Oct 29 22:26:46.532273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.532198    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"]
Oct 29 22:26:46.538169 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:26:46.538142    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100f773a_93c3_49ef_b704_393f72d7b181.slice/crio-e377b6a7d7b2804d560a659de6dc5c95a7538d6064bba22534dec494de00daf3 WatchSource:0}: Error finding container e377b6a7d7b2804d560a659de6dc5c95a7538d6064bba22534dec494de00daf3: Status 404 returned error can't find the container with id e377b6a7d7b2804d560a659de6dc5c95a7538d6064bba22534dec494de00daf3
Oct 29 22:26:46.606710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.606675    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:46.693165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.693036    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q" event={"ID":"100f773a-93c3-49ef-b704-393f72d7b181","Type":"ContainerStarted","Data":"8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931"}
Oct 29 22:26:46.693165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.693105    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q" event={"ID":"100f773a-93c3-49ef-b704-393f72d7b181","Type":"ContainerStarted","Data":"e377b6a7d7b2804d560a659de6dc5c95a7538d6064bba22534dec494de00daf3"}
Oct 29 22:26:46.693165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.693170    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"
Oct 29 22:26:46.729318 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.729289    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:46.739590 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.739546    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q" podStartSLOduration=1.7395344700000002 podStartE2EDuration="1.73953447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:46.737085855 +0000 UTC m=+748.761861988" watchObservedRunningTime="2026-10-29 22:26:46.73953447 +0000 UTC m=+748.764310601"
Oct 29 22:26:46.760780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.760746    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"]
Oct 29 22:26:46.764441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.764422    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"
Oct 29 22:26:46.798881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.798636    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-fc684b\""
Oct 29 22:26:46.799048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.798883    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-fc684b-dockercfg-qb7dp\""
Oct 29 22:26:46.800125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.800096    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"]
Oct 29 22:26:46.803748 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.803715    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c31608a-a11b-4927-ae4d-10c871e174e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn\" (UID: \"0c31608a-a11b-4927-ae4d-10c871e174e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"
Oct 29 22:26:46.803883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.803793    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5vz\" (UniqueName: \"kubernetes.io/projected/0c31608a-a11b-4927-ae4d-10c871e174e7-kube-api-access-kj5vz\") pod \"payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn\" (UID: \"0c31608a-a11b-4927-ae4d-10c871e174e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"
Oct 29 22:26:46.908924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.908876    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5vz\" (UniqueName: \"kubernetes.io/projected/0c31608a-a11b-4927-ae4d-10c871e174e7-kube-api-access-kj5vz\") pod \"payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn\" (UID: \"0c31608a-a11b-4927-ae4d-10c871e174e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"
Oct 29 22:26:46.909716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.909689    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c31608a-a11b-4927-ae4d-10c871e174e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn\" (UID: \"0c31608a-a11b-4927-ae4d-10c871e174e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"
Oct 29 22:26:46.911454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.910785    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c31608a-a11b-4927-ae4d-10c871e174e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn\" (UID: \"0c31608a-a11b-4927-ae4d-10c871e174e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"
Oct 29 22:26:46.929553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:46.929531    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5vz\" (UniqueName: \"kubernetes.io/projected/0c31608a-a11b-4927-ae4d-10c871e174e7-kube-api-access-kj5vz\") pod \"payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn\" (UID: \"0c31608a-a11b-4927-ae4d-10c871e174e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"
Oct 29 22:26:47.108854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:47.108773    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"
Oct 29 22:26:47.416301 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:26:47.416258    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c31608a_a11b_4927_ae4d_10c871e174e7.slice/crio-cca9c7c53ab4e0c4ad8aa8ac0dc1a3407965f1250098fb998ac71f5e87baa10e WatchSource:0}: Error finding container cca9c7c53ab4e0c4ad8aa8ac0dc1a3407965f1250098fb998ac71f5e87baa10e: Status 404 returned error can't find the container with id cca9c7c53ab4e0c4ad8aa8ac0dc1a3407965f1250098fb998ac71f5e87baa10e
Oct 29 22:26:47.416862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:47.416839    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"]
Oct 29 22:26:47.698909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:47.698871    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn" event={"ID":"0c31608a-a11b-4927-ae4d-10c871e174e7","Type":"ContainerStarted","Data":"c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294"}
Oct 29 22:26:47.699077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:47.698933    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn" event={"ID":"0c31608a-a11b-4927-ae4d-10c871e174e7","Type":"ContainerStarted","Data":"cca9c7c53ab4e0c4ad8aa8ac0dc1a3407965f1250098fb998ac71f5e87baa10e"}
Oct 29 22:26:47.699077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:47.698955    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"
Oct 29 22:26:47.743722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:47.743677    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn" podStartSLOduration=1.743662893 podStartE2EDuration="1.743662893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:47.742430056 +0000 UTC m=+749.767206199" watchObservedRunningTime="2026-10-29 22:26:47.743662893 +0000 UTC m=+749.768439023"
Oct 29 22:26:47.794087 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:47.794051    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:47.918132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:47.912050    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:48.054049 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:26:48.053970    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:26:48.299126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.299092    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:48.410877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.410777    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:48.551693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.551663    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"]
Oct 29 22:26:48.557299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.556756    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"
Oct 29 22:26:48.591347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.581910    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"]
Oct 29 22:26:48.595971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.595663    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2e046a\""
Oct 29 22:26:48.595971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.595831    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2e046a-dockercfg-pfzkk\""
Oct 29 22:26:48.627210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.627178    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvcl\" (UniqueName: \"kubernetes.io/projected/646e1a1e-8f19-4df7-91d8-4864270cae7d-kube-api-access-qjvcl\") pod \"payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg\" (UID: \"646e1a1e-8f19-4df7-91d8-4864270cae7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"
Oct 29 22:26:48.627357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.627236    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646e1a1e-8f19-4df7-91d8-4864270cae7d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg\" (UID: \"646e1a1e-8f19-4df7-91d8-4864270cae7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"
Oct 29 22:26:48.728480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.728448    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvcl\" (UniqueName: \"kubernetes.io/projected/646e1a1e-8f19-4df7-91d8-4864270cae7d-kube-api-access-qjvcl\") pod \"payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg\" (UID: \"646e1a1e-8f19-4df7-91d8-4864270cae7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"
Oct 29 22:26:48.728678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.728539    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646e1a1e-8f19-4df7-91d8-4864270cae7d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg\" (UID: \"646e1a1e-8f19-4df7-91d8-4864270cae7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"
Oct 29 22:26:48.729243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.729217    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646e1a1e-8f19-4df7-91d8-4864270cae7d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg\" (UID: \"646e1a1e-8f19-4df7-91d8-4864270cae7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"
Oct 29 22:26:48.753427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.753405    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvcl\" (UniqueName: \"kubernetes.io/projected/646e1a1e-8f19-4df7-91d8-4864270cae7d-kube-api-access-qjvcl\") pod \"payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg\" (UID: \"646e1a1e-8f19-4df7-91d8-4864270cae7d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"
Oct 29 22:26:48.888347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:48.887476    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"
Oct 29 22:26:49.196676 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:26:49.196648    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646e1a1e_8f19_4df7_91d8_4864270cae7d.slice/crio-7855383ab026276716bf881dcfecb73867695f0419d42980baa731bdbaef3dc9 WatchSource:0}: Error finding container 7855383ab026276716bf881dcfecb73867695f0419d42980baa731bdbaef3dc9: Status 404 returned error can't find the container with id 7855383ab026276716bf881dcfecb73867695f0419d42980baa731bdbaef3dc9
Oct 29 22:26:49.197858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.197834    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"]
Oct 29 22:26:49.232230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.232204    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:49.355006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.354983    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:49.396484 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.396454    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"]
Oct 29 22:26:49.400165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.400148    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"
Oct 29 22:26:49.416947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.416926    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-22d634\""
Oct 29 22:26:49.417047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.416973    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-22d634-dockercfg-smbzn\""
Oct 29 22:26:49.420811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.420790    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"]
Oct 29 22:26:49.436757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.436737    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96gvl\" (UniqueName: \"kubernetes.io/projected/601556d3-a9a1-43b8-88b7-320af479646d-kube-api-access-96gvl\") pod \"payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr\" (UID: \"601556d3-a9a1-43b8-88b7-320af479646d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"
Oct 29 22:26:49.436840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.436789    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/601556d3-a9a1-43b8-88b7-320af479646d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr\" (UID: \"601556d3-a9a1-43b8-88b7-320af479646d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"
Oct 29 22:26:49.503948 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.503877    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"]
Oct 29 22:26:49.507870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.507840    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"
Oct 29 22:26:49.511492 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.511467    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"]
Oct 29 22:26:49.538197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.538175    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35221c46-797c-4c8e-97a5-55d9bc6b4829-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p\" (UID: \"35221c46-797c-4c8e-97a5-55d9bc6b4829\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"
Oct 29 22:26:49.538369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.538319    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tvbc\" (UniqueName: \"kubernetes.io/projected/35221c46-797c-4c8e-97a5-55d9bc6b4829-kube-api-access-4tvbc\") pod \"payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p\" (UID: \"35221c46-797c-4c8e-97a5-55d9bc6b4829\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"
Oct 29 22:26:49.538483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.538410    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gvl\" (UniqueName: \"kubernetes.io/projected/601556d3-a9a1-43b8-88b7-320af479646d-kube-api-access-96gvl\") pod \"payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr\" (UID: \"601556d3-a9a1-43b8-88b7-320af479646d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"
Oct 29 22:26:49.538556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.538486    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/601556d3-a9a1-43b8-88b7-320af479646d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr\" (UID: \"601556d3-a9a1-43b8-88b7-320af479646d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"
Oct 29 22:26:49.539142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.539120    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/601556d3-a9a1-43b8-88b7-320af479646d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr\" (UID: \"601556d3-a9a1-43b8-88b7-320af479646d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"
Oct 29 22:26:49.564638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.564594    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96gvl\" (UniqueName: \"kubernetes.io/projected/601556d3-a9a1-43b8-88b7-320af479646d-kube-api-access-96gvl\") pod \"payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr\" (UID: \"601556d3-a9a1-43b8-88b7-320af479646d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"
Oct 29 22:26:49.639618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.639592    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35221c46-797c-4c8e-97a5-55d9bc6b4829-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p\" (UID: \"35221c46-797c-4c8e-97a5-55d9bc6b4829\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"
Oct 29 22:26:49.639767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.639668    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tvbc\" (UniqueName: \"kubernetes.io/projected/35221c46-797c-4c8e-97a5-55d9bc6b4829-kube-api-access-4tvbc\") pod \"payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p\" (UID: \"35221c46-797c-4c8e-97a5-55d9bc6b4829\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"
Oct 29 22:26:49.640280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.640261    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35221c46-797c-4c8e-97a5-55d9bc6b4829-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p\" (UID: \"35221c46-797c-4c8e-97a5-55d9bc6b4829\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"
Oct 29 22:26:49.664317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.663981    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tvbc\" (UniqueName: \"kubernetes.io/projected/35221c46-797c-4c8e-97a5-55d9bc6b4829-kube-api-access-4tvbc\") pod \"payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p\" (UID: \"35221c46-797c-4c8e-97a5-55d9bc6b4829\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"
Oct 29 22:26:49.713150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.713119    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"
Oct 29 22:26:49.719965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.719931    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg" event={"ID":"646e1a1e-8f19-4df7-91d8-4864270cae7d","Type":"ContainerStarted","Data":"f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5"}
Oct 29 22:26:49.720076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.719998    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"
Oct 29 22:26:49.720076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.720025    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg" event={"ID":"646e1a1e-8f19-4df7-91d8-4864270cae7d","Type":"ContainerStarted","Data":"7855383ab026276716bf881dcfecb73867695f0419d42980baa731bdbaef3dc9"}
Oct 29 22:26:49.731788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.731766    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:49.767033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.766974    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg" podStartSLOduration=1.7669538729999998 podStartE2EDuration="1.766953873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:49.765117332 +0000 UTC m=+751.789893464" watchObservedRunningTime="2026-10-29 22:26:49.766953873 +0000 UTC m=+751.791730005"
Oct 29 22:26:49.821758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.821731    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"
Oct 29 22:26:49.864243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:49.864206    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:50.027560 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:26:50.027528    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601556d3_a9a1_43b8_88b7_320af479646d.slice/crio-50c53af52856284a9dd0e9e1c19827ec5b761c59955385a5f169f6f2ff0fe271 WatchSource:0}: Error finding container 50c53af52856284a9dd0e9e1c19827ec5b761c59955385a5f169f6f2ff0fe271: Status 404 returned error can't find the container with id 50c53af52856284a9dd0e9e1c19827ec5b761c59955385a5f169f6f2ff0fe271
Oct 29 22:26:50.029126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:50.029100    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"]
Oct 29 22:26:50.146468 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:26:50.146093    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35221c46_797c_4c8e_97a5_55d9bc6b4829.slice/crio-bdc455d5bddb40a408827ab8e59141173dcc2c2803219d69897430c8329853ce WatchSource:0}: Error finding container bdc455d5bddb40a408827ab8e59141173dcc2c2803219d69897430c8329853ce: Status 404 returned error can't find the container with id bdc455d5bddb40a408827ab8e59141173dcc2c2803219d69897430c8329853ce
Oct 29 22:26:50.147347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:50.147305    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"]
Oct 29 22:26:50.741475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:50.741432    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p" event={"ID":"35221c46-797c-4c8e-97a5-55d9bc6b4829","Type":"ContainerStarted","Data":"c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559"}
Oct 29 22:26:50.742037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:50.741534    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p" event={"ID":"35221c46-797c-4c8e-97a5-55d9bc6b4829","Type":"ContainerStarted","Data":"bdc455d5bddb40a408827ab8e59141173dcc2c2803219d69897430c8329853ce"}
Oct 29 22:26:50.742037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:50.741567    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"
Oct 29 22:26:50.743566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:50.743539    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr" event={"ID":"601556d3-a9a1-43b8-88b7-320af479646d","Type":"ContainerStarted","Data":"d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373"}
Oct 29 22:26:50.743691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:50.743589    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr" event={"ID":"601556d3-a9a1-43b8-88b7-320af479646d","Type":"ContainerStarted","Data":"50c53af52856284a9dd0e9e1c19827ec5b761c59955385a5f169f6f2ff0fe271"}
Oct 29 22:26:50.743771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:50.743718    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"
Oct 29 22:26:50.798689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:50.798626    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p" podStartSLOduration=1.798607335 podStartE2EDuration="1.798607335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:50.795522203 +0000 UTC m=+752.820298352" watchObservedRunningTime="2026-10-29 22:26:50.798607335 +0000 UTC m=+752.823383467"
Oct 29 22:26:50.861290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:50.861242    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr" podStartSLOduration=1.861212939 podStartE2EDuration="1.861212939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:50.858539392 +0000 UTC m=+752.883315547" watchObservedRunningTime="2026-10-29 22:26:50.861212939 +0000 UTC m=+752.885989070"
Oct 29 22:26:51.548695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:51.548664    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:51.691367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:51.690275    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:51.815429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:51.815242    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:52.648457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:52.648429    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"
Oct 29 22:26:52.648654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:52.648508    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"
Oct 29 22:26:53.417888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:53.417852    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:53.542153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:53.542123    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:53.655786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:53.655746    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:53.656458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:53.656438    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:26:53.656556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:53.656498    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:26:54.664477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:54.664450    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"
Oct 29 22:26:54.664940 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:54.664528    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"
Oct 29 22:26:55.193980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:55.193939    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:56.697099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:56.697063    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"
Oct 29 22:26:56.801481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:56.801420    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:56.812341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:56.811748    4860 generic.go:356] "Generic (PLEG): container finished" podID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerID="53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c" exitCode=0
Oct 29 22:26:56.812341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:56.811885    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" event={"ID":"bbc22e15-7269-4307-a9bf-ad39641f7287","Type":"ContainerDied","Data":"53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c"}
Oct 29 22:26:56.812573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:56.812556    4860 scope.go:122] "RemoveContainer" containerID="53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c"
Oct 29 22:26:56.814198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:56.814162    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:26:56.814297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:56.814255    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:26:56.814400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:56.814306    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:26:56.931912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:56.931880    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:57.061396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:57.060665    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:57.829374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:57.828535    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" event={"ID":"bbc22e15-7269-4307-a9bf-ad39641f7287","Type":"ContainerStarted","Data":"6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0"}
Oct 29 22:26:57.845595 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:57.845567    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:26:57.845801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:57.845779    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:26:57.845857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:57.845837    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 29 22:26:58.065469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:58.065441    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:58.660502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:58.660461    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:58.715909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:58.715880    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"
Oct 29 22:26:58.716094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:58.715950    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"
Oct 29 22:26:58.799790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:58.799752    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:58.846794 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:58.846369    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:26:58.846794 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:58.846439    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 29 22:26:58.929262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:58.929175    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:59.094904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:59.094859    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:26:59.845996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:59.845910    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:26:59.846163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:26:59.846003    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 29 22:27:00.746585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:00.746552    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"
Oct 29 22:27:00.846525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:00.846495    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:27:00.846681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:00.846554    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 29 22:27:01.064903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.064824    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:01.401108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.401024    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"]
Oct 29 22:27:01.404597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.404575    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.425993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.425972    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"]
Oct 29 22:27:01.436548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.436515    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"simulator-w6-784e7a-kserve-self-signed-certs\""
Oct 29 22:27:01.436664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.436598    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"default-dockercfg-d8kph\""
Oct 29 22:27:01.484601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.484568    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"kube-root-ca.crt\""
Oct 29 22:27:01.492548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.492520    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-784e7a\"/\"openshift-service-ca.crt\""
Oct 29 22:27:01.492833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.492805    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:01.512046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.512020    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tmp-dir\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.512175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.512081    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmfc\" (UniqueName: \"kubernetes.io/projected/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-kube-api-access-jqmfc\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.512175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.512161    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-home\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.512253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.512185    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-dshm\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.512253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.512240    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tls-certs\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.512316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.512277    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-model-cache\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.613059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.613022    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmfc\" (UniqueName: \"kubernetes.io/projected/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-kube-api-access-jqmfc\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.613059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.613061    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-home\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.615376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.613082    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-dshm\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.615376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.613104    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tls-certs\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.615376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.613141    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-model-cache\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.615376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.613256    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tmp-dir\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.615376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.613592    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-home\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.615376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.613668    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tmp-dir\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.615376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.613672    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-model-cache\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.615599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.615433    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-dshm\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.615698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.615680    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tls-certs\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.640155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.640130    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmfc\" (UniqueName: \"kubernetes.io/projected/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-kube-api-access-jqmfc\") pod \"simulator-w6-784e7a-kserve-6f9d798769-glghx\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") " pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.718717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.718683    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:01.752267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.752246    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"
Oct 29 22:27:01.752674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.752348    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"
Oct 29 22:27:01.848930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.848824    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:27:01.848930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.848887    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 29 22:27:01.893351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.891853    4860 generic.go:356] "Generic (PLEG): container finished" podID="da28397d-fefc-4a03-8dc8-3fad87558566" containerID="ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61" exitCode=0
Oct 29 22:27:01.893351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.892012    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" event={"ID":"da28397d-fefc-4a03-8dc8-3fad87558566","Type":"ContainerDied","Data":"ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61"}
Oct 29 22:27:01.893351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.892515    4860 scope.go:122] "RemoveContainer" containerID="ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61"
Oct 29 22:27:01.894440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.894407    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:27:01.894611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.894595    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:27:01.894693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:01.894683    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:27:02.053638 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:27:02.053605    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7e09943_de5c_4a6d_8cf0_5ebd6ac32f0d.slice/crio-2b86168baaa9c5a62293a819f55e404202bdc6526bdf9d8b46e1d424d1ad60c0 WatchSource:0}: Error finding container 2b86168baaa9c5a62293a819f55e404202bdc6526bdf9d8b46e1d424d1ad60c0: Status 404 returned error can't find the container with id 2b86168baaa9c5a62293a819f55e404202bdc6526bdf9d8b46e1d424d1ad60c0
Oct 29 22:27:02.056459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:02.056431    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"]
Oct 29 22:27:02.845785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:02.845751    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 29 22:27:02.846266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:02.845812    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 29 22:27:02.909609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:02.909572    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" event={"ID":"da28397d-fefc-4a03-8dc8-3fad87558566","Type":"ContainerStarted","Data":"7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece"}
Oct 29 22:27:02.912219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:02.912182    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" event={"ID":"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d","Type":"ContainerStarted","Data":"4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42"}
Oct 29 22:27:02.912219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:02.912217    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" event={"ID":"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d","Type":"ContainerStarted","Data":"2b86168baaa9c5a62293a819f55e404202bdc6526bdf9d8b46e1d424d1ad60c0"}
Oct 29 22:27:03.002976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:03.001558    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:03.004803 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:03.004750    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podStartSLOduration=2.004736651 podStartE2EDuration="2.004736651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:03.002756372 +0000 UTC m=+765.027532501" watchObservedRunningTime="2026-10-29 22:27:03.004736651 +0000 UTC m=+765.029512785"
Oct 29 22:27:03.281381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:03.277496    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:27:03.281381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:03.279195    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:27:03.281381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:03.279246    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 29 22:27:03.718863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:03.718827    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:03.850313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:03.850280    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:27:03.850759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:03.850575    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:27:03.851393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:03.851374    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 22:27:04.286348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:04.279636    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:27:04.286348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:04.279692    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 29 22:27:04.421190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:04.421155    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:04.859278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:04.859244    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:05.018347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:05.018260    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:05.143058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:05.142979    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:05.285347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:05.282775    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:27:05.285347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:05.282836    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 29 22:27:05.421084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:05.420998    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:06.289348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:06.288687    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:27:06.289348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:06.288747    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 29 22:27:06.715137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:06.715098    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:07.291968 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:07.291421    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:27:07.291968 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:07.291482    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 29 22:27:07.615305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:07.614424    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:27:08.290812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:08.290777    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 29 22:27:08.291083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:08.291055    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 29 22:27:08.608446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:08.608357    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:09.294010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:09.293974    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:27:09.294920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:09.294891    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 22:27:09.447901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:09.447854    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:09.562510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:09.562427    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:09.673753 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:09.673720    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:10.069831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:10.069800    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:10.223295 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:10.223261    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:10.359475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:10.359390    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:10.499867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:10.499834    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:10.699394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:10.699346    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:11.719824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:11.719788    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:11.733172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:11.733145    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:11.997858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:11.997773    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:27:12.159685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.159653    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"]
Oct 29 22:27:12.165942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.165919    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.177371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.177348    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"]
Oct 29 22:27:12.182179 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.182155    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fc684b\"/\"openshift-service-ca.crt\""
Oct 29 22:27:12.182293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.182192    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fc684b\"/\"kube-root-ca.crt\""
Oct 29 22:27:12.182293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.182246    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fc684b\"/\"simulator-w1-fc684b-kserve-self-signed-certs\""
Oct 29 22:27:12.182293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.182262    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fc684b\"/\"default-dockercfg-7rdkc\""
Oct 29 22:27:12.233917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.233883    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:12.275030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.274964    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-model-cache\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.275030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.275016    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-home\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.275240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.275046    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a9daa1-d068-4a2e-8582-323349242bc5-tls-certs\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.275240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.275167    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-dshm\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.275379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.275281    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6c7j\" (UniqueName: \"kubernetes.io/projected/25a9daa1-d068-4a2e-8582-323349242bc5-kube-api-access-r6c7j\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.275379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.275316    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-tmp-dir\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.376183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.376154    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-home\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.376372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.376197    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a9daa1-d068-4a2e-8582-323349242bc5-tls-certs\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.376372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.376219    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-dshm\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.376372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.376284    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6c7j\" (UniqueName: \"kubernetes.io/projected/25a9daa1-d068-4a2e-8582-323349242bc5-kube-api-access-r6c7j\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.376372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.376311    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-tmp-dir\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.376372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.376366    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-model-cache\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.376846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.376809    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-tmp-dir\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.376954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.376824    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-home\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.376954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.376863    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-model-cache\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.378709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.378687    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-dshm\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.378782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.378755    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a9daa1-d068-4a2e-8582-323349242bc5-tls-certs\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.399003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.398968    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6c7j\" (UniqueName: \"kubernetes.io/projected/25a9daa1-d068-4a2e-8582-323349242bc5-kube-api-access-r6c7j\") pod \"simulator-w1-fc684b-kserve-78554888bf-8fc7d\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") " pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.483766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.483729    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:12.572470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.572441    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:12.796002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.795919    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"]
Oct 29 22:27:12.988587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.988544    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" event={"ID":"25a9daa1-d068-4a2e-8582-323349242bc5","Type":"ContainerStarted","Data":"a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433"}
Oct 29 22:27:12.988771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:12.988602    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" event={"ID":"25a9daa1-d068-4a2e-8582-323349242bc5","Type":"ContainerStarted","Data":"435427e69ca600be5f924ddb5a9e06c1bb03d44aefbf397556cd5cc5b149af6c"}
Oct 29 22:27:13.035164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:13.035119    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podStartSLOduration=1.035105886 podStartE2EDuration="1.035105886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:13.03225275 +0000 UTC m=+775.057028888" watchObservedRunningTime="2026-10-29 22:27:13.035105886 +0000 UTC m=+775.059882115"
Oct 29 22:27:13.484898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:13.484833    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:13.997461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:13.997428    4860 generic.go:356] "Generic (PLEG): container finished" podID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerID="59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00" exitCode=0
Oct 29 22:27:13.997850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:13.997501    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" event={"ID":"ec041e3f-8c83-4807-94f2-40a0636a4dd9","Type":"ContainerDied","Data":"59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00"}
Oct 29 22:27:13.997946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:13.997930    4860 scope.go:122] "RemoveContainer" containerID="59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00"
Oct 29 22:27:13.999136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:13.999110    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:27:13.999195 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:13.999175    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:27:13.999229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:13.999221    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:27:14.353671 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:14.353580    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:14.616855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:14.616772    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:15.005457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:15.005420    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" event={"ID":"ec041e3f-8c83-4807-94f2-40a0636a4dd9","Type":"ContainerStarted","Data":"92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0"}
Oct 29 22:27:15.201313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:15.201277    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:15.323517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:15.323435    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:15.451966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:15.451938    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:15.539744 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:15.539703    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:27:15.539988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:15.539968    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:27:15.540040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:15.540012    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 29 22:27:16.293117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:16.293082    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:16.539890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:16.539856    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:27:16.540053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:16.539905    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 29 22:27:17.090562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:17.090529    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:17.539618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:17.539581    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:27:17.540095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:17.539641    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 29 22:27:17.950895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:17.950859    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:18.056794 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:27:18.056754    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:27:18.225715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.225640    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"]
Oct 29 22:27:18.231013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.230991    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.252757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.252729    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"]
Oct 29 22:27:18.262787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.261782    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"kube-root-ca.crt\""
Oct 29 22:27:18.262787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.262011    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"default-dockercfg-k2fzk\""
Oct 29 22:27:18.263055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.263038    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"simulator-w3-43b37d-kserve-self-signed-certs\""
Oct 29 22:27:18.263225 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.263206    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"openshift-service-ca.crt\""
Oct 29 22:27:18.336714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.336683    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-home\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.336889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.336723    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pgqt\" (UniqueName: \"kubernetes.io/projected/58e4fb66-49ec-436a-8e2a-8890f83ed081-kube-api-access-2pgqt\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.336889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.336832    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-model-cache\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.337010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.336932    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-tmp-dir\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.337010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.336977    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e4fb66-49ec-436a-8e2a-8890f83ed081-tls-certs\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.337120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.337049    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-dshm\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.337944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.337917    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:18.437790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.437754    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-home\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.437790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.437788    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pgqt\" (UniqueName: \"kubernetes.io/projected/58e4fb66-49ec-436a-8e2a-8890f83ed081-kube-api-access-2pgqt\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.438040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.437834    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-model-cache\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.438040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.437893    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-tmp-dir\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.438040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.438002    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e4fb66-49ec-436a-8e2a-8890f83ed081-tls-certs\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.438207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.438054    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-dshm\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.438267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.438207    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-home\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.438267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.438229    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-model-cache\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.438402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.438316    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-tmp-dir\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.440379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.440356    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-dshm\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.440667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.440646    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e4fb66-49ec-436a-8e2a-8890f83ed081-tls-certs\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.458145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.458117    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pgqt\" (UniqueName: \"kubernetes.io/projected/58e4fb66-49ec-436a-8e2a-8890f83ed081-kube-api-access-2pgqt\") pod \"simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") " pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.539310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.539219    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:27:18.539310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.539285    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 29 22:27:18.551786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.551755    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:18.910995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:18.906893    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"]
Oct 29 22:27:18.912589 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:27:18.912561    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58e4fb66_49ec_436a_8e2a_8890f83ed081.slice/crio-f42a5ecad1584cd50cd027641e66178a332dc0b6e335584ca4f82a9845173f9d WatchSource:0}: Error finding container f42a5ecad1584cd50cd027641e66178a332dc0b6e335584ca4f82a9845173f9d: Status 404 returned error can't find the container with id f42a5ecad1584cd50cd027641e66178a332dc0b6e335584ca4f82a9845173f9d
Oct 29 22:27:19.037207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:19.036945    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" event={"ID":"58e4fb66-49ec-436a-8e2a-8890f83ed081","Type":"ContainerStarted","Data":"4a8dedc61c05bdbeb3d0547296b96802cab77c456df133b67650cceef06aa582"}
Oct 29 22:27:19.037207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:19.036996    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" event={"ID":"58e4fb66-49ec-436a-8e2a-8890f83ed081","Type":"ContainerStarted","Data":"f42a5ecad1584cd50cd027641e66178a332dc0b6e335584ca4f82a9845173f9d"}
Oct 29 22:27:19.092348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:19.090522    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podStartSLOduration=1.090503409 podStartE2EDuration="1.090503409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:19.087353415 +0000 UTC m=+781.112129570" watchObservedRunningTime="2026-10-29 22:27:19.090503409 +0000 UTC m=+781.115279541"
Oct 29 22:27:19.539833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:19.539795    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 29 22:27:19.540036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:19.539859    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 29 22:27:19.552866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:19.552834    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:19.625481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:19.625451    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:19.756974 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:19.756937    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:20.366138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:20.366102    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:20.543385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:20.543359    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:27:20.543771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:20.543747    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:27:20.544407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:20.544385    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:20.544596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:20.544582    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 22:27:20.677799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:20.677720    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:21.545414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:21.545380    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:21.974842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:21.974806    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:22.489367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:22.485463    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:22.509788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:22.509742    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:23.080187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:23.080158    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:27:23.259694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:23.259664    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:24.244295 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.244263    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"]
Oct 29 22:27:24.249920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.249897    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.262655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.262634    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"]
Oct 29 22:27:24.280105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.280083    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-22d634\"/\"openshift-service-ca.crt\""
Oct 29 22:27:24.280105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.280083    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-22d634\"/\"simulator-w5-22d634-kserve-self-signed-certs\""
Oct 29 22:27:24.280286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.280123    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-22d634\"/\"default-dockercfg-x59xw\""
Oct 29 22:27:24.280286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.280130    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-22d634\"/\"kube-root-ca.crt\""
Oct 29 22:27:24.382056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.382035    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:24.398736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.398717    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5837adb2-8517-4f79-97c0-81535c4a9f8a-tls-certs\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.398826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.398746    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-model-cache\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.398826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.398778    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-dshm\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.398905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.398863    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-home\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.398941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.398925    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-tmp-dir\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.399005 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.398983    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pxxq\" (UniqueName: \"kubernetes.io/projected/5837adb2-8517-4f79-97c0-81535c4a9f8a-kube-api-access-2pxxq\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.500498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.500445    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5837adb2-8517-4f79-97c0-81535c4a9f8a-tls-certs\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.500498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.500477    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-model-cache\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.500624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.500568    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-dshm\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.500624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.500601    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-home\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.500688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.500669    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-tmp-dir\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.500727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.500702    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pxxq\" (UniqueName: \"kubernetes.io/projected/5837adb2-8517-4f79-97c0-81535c4a9f8a-kube-api-access-2pxxq\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.500794 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.500779    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-model-cache\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.500988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.500968    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-home\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.501092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.501025    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-tmp-dir\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.502716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.502698    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-dshm\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.503137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.503117    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5837adb2-8517-4f79-97c0-81535c4a9f8a-tls-certs\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.521489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.521464    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pxxq\" (UniqueName: \"kubernetes.io/projected/5837adb2-8517-4f79-97c0-81535c4a9f8a-kube-api-access-2pxxq\") pod \"simulator-w5-22d634-kserve-84ff4d869f-nx68n\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") " pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.563161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.563129    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:24.862913 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:27:24.862873    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5837adb2_8517_4f79_97c0_81535c4a9f8a.slice/crio-873025e2e93ffed3ca3b5a8d6e8a10d927fb90bf638c57251197c56f7e3b094e WatchSource:0}: Error finding container 873025e2e93ffed3ca3b5a8d6e8a10d927fb90bf638c57251197c56f7e3b094e: Status 404 returned error can't find the container with id 873025e2e93ffed3ca3b5a8d6e8a10d927fb90bf638c57251197c56f7e3b094e
Oct 29 22:27:24.864514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:24.864494    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"]
Oct 29 22:27:25.050429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:25.050399    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:25.078464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:25.078433    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" event={"ID":"5837adb2-8517-4f79-97c0-81535c4a9f8a","Type":"ContainerStarted","Data":"65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c"}
Oct 29 22:27:25.078596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:25.078477    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" event={"ID":"5837adb2-8517-4f79-97c0-81535c4a9f8a","Type":"ContainerStarted","Data":"873025e2e93ffed3ca3b5a8d6e8a10d927fb90bf638c57251197c56f7e3b094e"}
Oct 29 22:27:25.126442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:25.126308    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podStartSLOduration=1.12628906 podStartE2EDuration="1.12628906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:25.120361504 +0000 UTC m=+787.145137637" watchObservedRunningTime="2026-10-29 22:27:25.12628906 +0000 UTC m=+787.151065192"
Oct 29 22:27:25.563336 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:25.563304    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:26.717699 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:26.717649    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:27.868737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:27.868699    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:28.480835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:28.480805    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:28.552215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:28.552179    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:28.565757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:28.565728    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:29.116891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:29.116859    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:27:30.370060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:30.370028    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:32.123697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:32.123662    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:33.908630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:33.908597    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:34.563658 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:34.563625    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:34.575781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:34.575754    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:35.002842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.002812    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"]
Oct 29 22:27:35.006536 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.006517    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.024930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.024903    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"]
Oct 29 22:27:35.035625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.035597    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-08ebee\"/\"openshift-service-ca.crt\""
Oct 29 22:27:35.035728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.035605    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-08ebee\"/\"kube-root-ca.crt\""
Oct 29 22:27:35.036077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.036059    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-08ebee\"/\"default-dockercfg-pfpfc\""
Oct 29 22:27:35.036147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.036065    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-08ebee\"/\"simulator-w4-08ebee-kserve-self-signed-certs\""
Oct 29 22:27:35.077364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.077314    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:35.109482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.109455    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-home\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.109588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.109500    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-tmp-dir\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.109588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.109527    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-model-cache\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.109669 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.109655    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krddp\" (UniqueName: \"kubernetes.io/projected/3e631530-8524-445b-90da-4ff35c1c7f54-kube-api-access-krddp\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.109771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.109756    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e631530-8524-445b-90da-4ff35c1c7f54-tls-certs\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.109808 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.109795    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-dshm\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.148607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.148585    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:27:35.210726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.210696    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e631530-8524-445b-90da-4ff35c1c7f54-tls-certs\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.210847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.210763    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-dshm\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.210847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.210827    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-home\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.210959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.210871    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-tmp-dir\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.211034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.211009    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-model-cache\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.211089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.211078    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krddp\" (UniqueName: \"kubernetes.io/projected/3e631530-8524-445b-90da-4ff35c1c7f54-kube-api-access-krddp\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.211370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.211347    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-tmp-dir\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.211567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.211361    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-home\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.211567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.211481    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-model-cache\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.213547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.213526    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-dshm\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.213870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.213851    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e631530-8524-445b-90da-4ff35c1c7f54-tls-certs\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.231950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.231924    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krddp\" (UniqueName: \"kubernetes.io/projected/3e631530-8524-445b-90da-4ff35c1c7f54-kube-api-access-krddp\") pod \"simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") " pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.319700 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.319612    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:35.544521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.544476    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:35.626578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:35.626493    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"]
Oct 29 22:27:35.633768 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:27:35.633733    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e631530_8524_445b_90da_4ff35c1c7f54.slice/crio-d8527fedad6a6497427ebe78b01708d49a2f83e6685c45115538985f1374a033 WatchSource:0}: Error finding container d8527fedad6a6497427ebe78b01708d49a2f83e6685c45115538985f1374a033: Status 404 returned error can't find the container with id d8527fedad6a6497427ebe78b01708d49a2f83e6685c45115538985f1374a033
Oct 29 22:27:36.147740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:36.147694    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" event={"ID":"3e631530-8524-445b-90da-4ff35c1c7f54","Type":"ContainerStarted","Data":"fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f"}
Oct 29 22:27:36.148164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:36.147750    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" event={"ID":"3e631530-8524-445b-90da-4ff35c1c7f54","Type":"ContainerStarted","Data":"d8527fedad6a6497427ebe78b01708d49a2f83e6685c45115538985f1374a033"}
Oct 29 22:27:36.197771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:36.197709    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podStartSLOduration=2.197694646 podStartE2EDuration="2.197694646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:36.194432509 +0000 UTC m=+798.219208647" watchObservedRunningTime="2026-10-29 22:27:36.197694646 +0000 UTC m=+798.222470776"
Oct 29 22:27:36.320753 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:36.320719    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:37.260723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:37.260672    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:39.157349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:39.157291    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:39.374516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:39.374481    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:40.841264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:40.841221    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:42.671921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:42.671882    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:44.489302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:44.489260    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:45.320760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:45.320727    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:45.333804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:45.333780    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:46.189244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:46.189212    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:46.225634 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:46.225604    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:27:48.055670 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:27:48.055635    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:27:48.068234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:48.068206    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:49.850259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:49.850229    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:50.048421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.048393    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"]
Oct 29 22:27:50.053730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.053706    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.074025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.073998    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"]
Oct 29 22:27:50.080942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.080701    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fc684b\"/\"premium-w1-fc684b-kserve-self-signed-certs\""
Oct 29 22:27:50.136504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.136431    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:50.183080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.183050    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkzw4\" (UniqueName: \"kubernetes.io/projected/d2a78176-4066-475f-bb02-5f2ea982ebae-kube-api-access-jkzw4\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.183232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.183088    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-tmp-dir\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.183232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.183106    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-home\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.183334 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.183229    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a78176-4066-475f-bb02-5f2ea982ebae-tls-certs\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.183334 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.183265    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-dshm\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.183411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.183391    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-model-cache\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.284418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.284387    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkzw4\" (UniqueName: \"kubernetes.io/projected/d2a78176-4066-475f-bb02-5f2ea982ebae-kube-api-access-jkzw4\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.284619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.284430    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-tmp-dir\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.284619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.284447    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-home\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.284619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.284488    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a78176-4066-475f-bb02-5f2ea982ebae-tls-certs\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.284619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.284515    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-dshm\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.284619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.284578    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-model-cache\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.284901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.284815    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-tmp-dir\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.284901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.284875    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-home\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.284977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.284934    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-model-cache\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.286897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.286873    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-dshm\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.287169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.287153    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a78176-4066-475f-bb02-5f2ea982ebae-tls-certs\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.304949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.304918    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkzw4\" (UniqueName: \"kubernetes.io/projected/d2a78176-4066-475f-bb02-5f2ea982ebae-kube-api-access-jkzw4\") pod \"premium-w1-fc684b-kserve-5c4df557b-6bwkz\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") " pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.390846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.390778    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:50.688625 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:27:50.688599    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2a78176_4066_475f_bb02_5f2ea982ebae.slice/crio-9424e3196c9bb6d85cf4f73166cdb11ad9689a1baf0d94a3dccd537e98c0aea9 WatchSource:0}: Error finding container 9424e3196c9bb6d85cf4f73166cdb11ad9689a1baf0d94a3dccd537e98c0aea9: Status 404 returned error can't find the container with id 9424e3196c9bb6d85cf4f73166cdb11ad9689a1baf0d94a3dccd537e98c0aea9
Oct 29 22:27:50.690368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:50.690343    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"]
Oct 29 22:27:51.247688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:51.247650    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" event={"ID":"d2a78176-4066-475f-bb02-5f2ea982ebae","Type":"ContainerStarted","Data":"66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb"}
Oct 29 22:27:51.247688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:51.247690    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" event={"ID":"d2a78176-4066-475f-bb02-5f2ea982ebae","Type":"ContainerStarted","Data":"9424e3196c9bb6d85cf4f73166cdb11ad9689a1baf0d94a3dccd537e98c0aea9"}
Oct 29 22:27:51.290157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:51.290108    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podStartSLOduration=1.290095301 podStartE2EDuration="1.290095301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:51.288408493 +0000 UTC m=+813.313184625" watchObservedRunningTime="2026-10-29 22:27:51.290095301 +0000 UTC m=+813.314871432"
Oct 29 22:27:51.391048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:51.391019    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:27:51.469816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:51.469776    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:53.169373 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:53.169320    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:55.038946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:55.038904    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:56.239855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:56.239822    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:56.880828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:56.880769    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:58.667877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:58.667821    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:27:59.216859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:27:59.216812    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:00.391590 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:00.391563    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:28:00.404380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:00.404357    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:28:00.480544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:00.480508    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:01.318460 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:01.318434    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:28:02.458697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:02.458660    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:04.167102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:04.167057    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:05.176388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:05.176355    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:05.841992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:05.841958    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:07.756099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:07.756068    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:09.492849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:09.492817    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:10.183388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.183355    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"]
Oct 29 22:28:10.187399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.187376    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.214310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.214277    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:10.221408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.221381    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"kube-root-ca.crt\""
Oct 29 22:28:10.221408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.221383    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"openshift-service-ca.crt\""
Oct 29 22:28:10.221630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.221430    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"premium-w2-2ba10a-kserve-self-signed-certs\""
Oct 29 22:28:10.221630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.221504    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"default-dockercfg-56tcf\""
Oct 29 22:28:10.224308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.224287    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"]
Oct 29 22:28:10.285069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.285042    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tmp-dir\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.285227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.285211    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tls-certs\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.285269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.285249    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtwlw\" (UniqueName: \"kubernetes.io/projected/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-kube-api-access-wtwlw\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.285319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.285304    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-home\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.285492 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.285473    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-dshm\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.285668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.285650    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-model-cache\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.349320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.349292    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:10.386984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.386953    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-model-cache\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.402381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.387050    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tmp-dir\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.402381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.387116    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tls-certs\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.402381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.387151    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtwlw\" (UniqueName: \"kubernetes.io/projected/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-kube-api-access-wtwlw\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.402381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.387227    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-home\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.402381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.387286    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-dshm\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.402381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.388674    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-model-cache\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.402381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.388826    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-home\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.402381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.388946    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tmp-dir\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.402381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.396145    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tls-certs\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.407468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.407440    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-dshm\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.411727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.411701    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtwlw\" (UniqueName: \"kubernetes.io/projected/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-kube-api-access-wtwlw\") pod \"premium-w2-2ba10a-kserve-79f58b59f9-cf5bh\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") " pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.500489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.500428    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:10.805120 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:28:10.805093    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d99f80b_82a0_4f37_b1e2_8d85a6ba1735.slice/crio-874e83521460b2f4d4b55296a03c4e218ff0ca9dfba380dc6f3fa30eeadd8432 WatchSource:0}: Error finding container 874e83521460b2f4d4b55296a03c4e218ff0ca9dfba380dc6f3fa30eeadd8432: Status 404 returned error can't find the container with id 874e83521460b2f4d4b55296a03c4e218ff0ca9dfba380dc6f3fa30eeadd8432
Oct 29 22:28:10.805256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:10.805235    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"]
Oct 29 22:28:11.373531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:11.373493    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" event={"ID":"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735","Type":"ContainerStarted","Data":"0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0"}
Oct 29 22:28:11.373677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:11.373537    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" event={"ID":"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735","Type":"ContainerStarted","Data":"874e83521460b2f4d4b55296a03c4e218ff0ca9dfba380dc6f3fa30eeadd8432"}
Oct 29 22:28:11.420595 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:11.420551    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podStartSLOduration=1.420539047 podStartE2EDuration="1.420539047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:11.41864468 +0000 UTC m=+833.443420810" watchObservedRunningTime="2026-10-29 22:28:11.420539047 +0000 UTC m=+833.445315177"
Oct 29 22:28:11.478139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:11.478111    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:11.501140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:11.501117    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:11.934523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:11.934487    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:13.209946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:13.209916    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:14.955721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:14.955684    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:15.432747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:15.432701    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:16.868974 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:16.868940    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:18.053974 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:28:18.053931    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:28:18.540314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:18.540278    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:20.253277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:20.253243    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:20.501551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:20.501517    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:20.514137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:20.514071    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:21.448868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:21.448840    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:28:21.883891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:21.883811    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:23.651615 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:23.651577    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:25.353790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:25.353759    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:26.240017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.239985    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"]
Oct 29 22:28:26.245219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.245196    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:26.265655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.265632    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"]
Oct 29 22:28:26.275516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.275367    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:28:26.275516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.275381    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:28:26.275516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.275477    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:28:26.276540 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.276516    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:28:26.354570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.354545    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cffa770-4cv98\" (UID: \"215017aa-3527-489e-9ca4-aa473df6746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:26.354898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.354709    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgqnz\" (UniqueName: \"kubernetes.io/projected/215017aa-3527-489e-9ca4-aa473df6746a-kube-api-access-vgqnz\") pod \"maas-api-revoke-sub-3cffa770-4cv98\" (UID: \"215017aa-3527-489e-9ca4-aa473df6746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:26.354898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.354759    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cffa770-4cv98\" (UID: \"215017aa-3527-489e-9ca4-aa473df6746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:26.455733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.455708    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgqnz\" (UniqueName: \"kubernetes.io/projected/215017aa-3527-489e-9ca4-aa473df6746a-kube-api-access-vgqnz\") pod \"maas-api-revoke-sub-3cffa770-4cv98\" (UID: \"215017aa-3527-489e-9ca4-aa473df6746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:26.455864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.455761    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cffa770-4cv98\" (UID: \"215017aa-3527-489e-9ca4-aa473df6746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:26.455864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.455805    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cffa770-4cv98\" (UID: \"215017aa-3527-489e-9ca4-aa473df6746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:26.456515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.456495    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cffa770-4cv98\" (UID: \"215017aa-3527-489e-9ca4-aa473df6746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:26.458083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.458062    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cffa770-4cv98\" (UID: \"215017aa-3527-489e-9ca4-aa473df6746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:26.477537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.477513    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgqnz\" (UniqueName: \"kubernetes.io/projected/215017aa-3527-489e-9ca4-aa473df6746a-kube-api-access-vgqnz\") pod \"maas-api-revoke-sub-3cffa770-4cv98\" (UID: \"215017aa-3527-489e-9ca4-aa473df6746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:26.588441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.588387    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:26.884704 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:28:26.884677    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215017aa_3527_489e_9ca4_aa473df6746a.slice/crio-8996d60aaa9b0848f25bac00dfe22848e12ee9e85a77a0728248ae0335b88ebd WatchSource:0}: Error finding container 8996d60aaa9b0848f25bac00dfe22848e12ee9e85a77a0728248ae0335b88ebd: Status 404 returned error can't find the container with id 8996d60aaa9b0848f25bac00dfe22848e12ee9e85a77a0728248ae0335b88ebd
Oct 29 22:28:26.884967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.884947    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"]
Oct 29 22:28:26.886594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:26.886580    4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:28:27.342076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:27.342045    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:27.477233 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:27.477200    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98" event={"ID":"215017aa-3527-489e-9ca4-aa473df6746a","Type":"ContainerStarted","Data":"8996d60aaa9b0848f25bac00dfe22848e12ee9e85a77a0728248ae0335b88ebd"}
Oct 29 22:28:29.179216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:29.179164    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:30.499312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:30.499226    4860 generic.go:356] "Generic (PLEG): container finished" podID="215017aa-3527-489e-9ca4-aa473df6746a" containerID="48f44e124026fbf466cde28bea93bdbfe5dea84f5075b3f102b69a3600ae3bce" exitCode=0
Oct 29 22:28:30.499820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:30.499310    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98" event={"ID":"215017aa-3527-489e-9ca4-aa473df6746a","Type":"ContainerDied","Data":"48f44e124026fbf466cde28bea93bdbfe5dea84f5075b3f102b69a3600ae3bce"}
Oct 29 22:28:31.244880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:31.244843    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:31.920515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:31.920486    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:32.084620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:32.084579    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-xks-ca\") pod \"215017aa-3527-489e-9ca4-aa473df6746a\" (UID: \"215017aa-3527-489e-9ca4-aa473df6746a\") "
Oct 29 22:28:32.084818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:32.084677    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/215017aa-3527-489e-9ca4-aa473df6746a-kube-api-access-vgqnz\" (UniqueName: \"kubernetes.io/projected/215017aa-3527-489e-9ca4-aa473df6746a-kube-api-access-vgqnz\") pod \"215017aa-3527-489e-9ca4-aa473df6746a\" (UID: \"215017aa-3527-489e-9ca4-aa473df6746a\") "
Oct 29 22:28:32.084818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:32.084732    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-openshift-ca\") pod \"215017aa-3527-489e-9ca4-aa473df6746a\" (UID: \"215017aa-3527-489e-9ca4-aa473df6746a\") "
Oct 29 22:28:32.085172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:32.085143    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-openshift-ca" pod "215017aa-3527-489e-9ca4-aa473df6746a" (UID: "215017aa-3527-489e-9ca4-aa473df6746a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:28:32.086936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:32.086899    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-xks-ca" pod "215017aa-3527-489e-9ca4-aa473df6746a" (UID: "215017aa-3527-489e-9ca4-aa473df6746a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:28:32.087042 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:32.087009    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215017aa-3527-489e-9ca4-aa473df6746a-kube-api-access-vgqnz" pod "215017aa-3527-489e-9ca4-aa473df6746a" (UID: "215017aa-3527-489e-9ca4-aa473df6746a"). InnerVolumeSpecName "kube-api-access-vgqnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:28:32.186358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:32.186314    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:32.186358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:32.186358    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgqnz\" (UniqueName: \"kubernetes.io/projected/215017aa-3527-489e-9ca4-aa473df6746a-kube-api-access-vgqnz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:32.186531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:32.186368    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215017aa-3527-489e-9ca4-aa473df6746a-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:32.540159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:32.540070    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98" event={"ID":"215017aa-3527-489e-9ca4-aa473df6746a","Type":"ContainerDied","Data":"8996d60aaa9b0848f25bac00dfe22848e12ee9e85a77a0728248ae0335b88ebd"}
Oct 29 22:28:32.540159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:32.540104    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"
Oct 29 22:28:32.540159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:32.540120    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8996d60aaa9b0848f25bac00dfe22848e12ee9e85a77a0728248ae0335b88ebd"
Oct 29 22:28:33.216518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.216453    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:33.279315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.279286    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"]
Oct 29 22:28:33.279794 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.279782    4860 cpu_manager.go:414] "containerMap: removing container" podUID="215017aa-3527-489e-9ca4-aa473df6746a" containerName="revoke-keys"
Oct 29 22:28:33.279838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.279795    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="215017aa-3527-489e-9ca4-aa473df6746a" containerName="revoke-keys"
Oct 29 22:28:33.279893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.279885    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="215017aa-3527-489e-9ca4-aa473df6746a" containerName="revoke-keys"
Oct 29 22:28:33.283429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.283405    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:33.303752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.303729    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"]
Oct 29 22:28:33.313950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.313923    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:28:33.313950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.313925    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:28:33.314114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.313935    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:28:33.314114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.313970    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:28:33.397749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.397725    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqzq\" (UniqueName: \"kubernetes.io/projected/399fff50-fdaf-4acb-9805-ac08d8313079-kube-api-access-8zqzq\") pod \"maas-api-revoke-sub-8971ebef-q8tnw\" (UID: \"399fff50-fdaf-4acb-9805-ac08d8313079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:33.397867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.397754    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8971ebef-q8tnw\" (UID: \"399fff50-fdaf-4acb-9805-ac08d8313079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:33.397908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.397881    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8971ebef-q8tnw\" (UID: \"399fff50-fdaf-4acb-9805-ac08d8313079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:33.498632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.498566    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8971ebef-q8tnw\" (UID: \"399fff50-fdaf-4acb-9805-ac08d8313079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:33.498764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.498672    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqzq\" (UniqueName: \"kubernetes.io/projected/399fff50-fdaf-4acb-9805-ac08d8313079-kube-api-access-8zqzq\") pod \"maas-api-revoke-sub-8971ebef-q8tnw\" (UID: \"399fff50-fdaf-4acb-9805-ac08d8313079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:33.498764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.498694    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8971ebef-q8tnw\" (UID: \"399fff50-fdaf-4acb-9805-ac08d8313079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:33.499354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.499318    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8971ebef-q8tnw\" (UID: \"399fff50-fdaf-4acb-9805-ac08d8313079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:33.501012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.500986    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8971ebef-q8tnw\" (UID: \"399fff50-fdaf-4acb-9805-ac08d8313079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:33.521594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.521568    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqzq\" (UniqueName: \"kubernetes.io/projected/399fff50-fdaf-4acb-9805-ac08d8313079-kube-api-access-8zqzq\") pod \"maas-api-revoke-sub-8971ebef-q8tnw\" (UID: \"399fff50-fdaf-4acb-9805-ac08d8313079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:33.596685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.596662    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:33.900226 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:28:33.900197    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399fff50_fdaf_4acb_9805_ac08d8313079.slice/crio-89fee656881131f5078da8093b83b255543741971641259a99fab7e2b35d8de8 WatchSource:0}: Error finding container 89fee656881131f5078da8093b83b255543741971641259a99fab7e2b35d8de8: Status 404 returned error can't find the container with id 89fee656881131f5078da8093b83b255543741971641259a99fab7e2b35d8de8
Oct 29 22:28:33.902614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:33.902587    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"]
Oct 29 22:28:34.552715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:34.552684    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw" event={"ID":"399fff50-fdaf-4acb-9805-ac08d8313079","Type":"ContainerStarted","Data":"49ef0af0b6d622b116e625fe6cafacbcb5c7ae45727c4945a14faed3f4bb5563"}
Oct 29 22:28:34.553164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:34.552742    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw" event={"ID":"399fff50-fdaf-4acb-9805-ac08d8313079","Type":"ContainerStarted","Data":"89fee656881131f5078da8093b83b255543741971641259a99fab7e2b35d8de8"}
Oct 29 22:28:34.603800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:34.603759    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw" podStartSLOduration=1.603746873 podStartE2EDuration="1.603746873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:34.601623099 +0000 UTC m=+856.626399231" watchObservedRunningTime="2026-10-29 22:28:34.603746873 +0000 UTC m=+856.628523003"
Oct 29 22:28:35.077210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.077184    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"]
Oct 29 22:28:35.080792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.080776    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.099362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.098642    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-43b37d-openshift-default-dockercfg-g742w\""
Oct 29 22:28:35.120214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.120179    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"]
Oct 29 22:28:35.130265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.130233    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:35.215623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.215595    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.215754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.215631    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.215754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.215649    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.215867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.215811    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlqr9\" (UniqueName: \"kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-kube-api-access-hlqr9\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.215963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.215942    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.216026 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.216014    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.216172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.216153    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.216256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.216189    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb326055-bb08-42f0-8597-ddfe942b2ca5-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.216256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.216243    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.317764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.317741    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlqr9\" (UniqueName: \"kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-kube-api-access-hlqr9\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.317881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.317788    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.317881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.317811    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.317881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.317827    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.317881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.317841    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb326055-bb08-42f0-8597-ddfe942b2ca5-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.317881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.317868    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.318144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.317941    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.318144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.317969    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.318252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.318159    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.318380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.318356    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.318457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.318375    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.318457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.318422    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.318575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.318558    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.318990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.318970    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb326055-bb08-42f0-8597-ddfe942b2ca5-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.320483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.320459    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.320591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.320558    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.345200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.345145    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlqr9\" (UniqueName: \"kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-kube-api-access-hlqr9\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.345200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.345172    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.398189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.398167    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:35.561237 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.561203    4860 generic.go:356] "Generic (PLEG): container finished" podID="399fff50-fdaf-4acb-9805-ac08d8313079" containerID="49ef0af0b6d622b116e625fe6cafacbcb5c7ae45727c4945a14faed3f4bb5563" exitCode=0
Oct 29 22:28:35.561607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.561292    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw" event={"ID":"399fff50-fdaf-4acb-9805-ac08d8313079","Type":"ContainerDied","Data":"49ef0af0b6d622b116e625fe6cafacbcb5c7ae45727c4945a14faed3f4bb5563"}
Oct 29 22:28:35.717821 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:28:35.717792    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb326055_bb08_42f0_8597_ddfe942b2ca5.slice/crio-081aa9aaf4b49e9cfc942d5b182331e877f888c78dc4a697cc6092cb034df64d WatchSource:0}: Error finding container 081aa9aaf4b49e9cfc942d5b182331e877f888c78dc4a697cc6092cb034df64d: Status 404 returned error can't find the container with id 081aa9aaf4b49e9cfc942d5b182331e877f888c78dc4a697cc6092cb034df64d
Oct 29 22:28:35.720726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.720690    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:28:35.720837 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.720769    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:28:35.720837 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.720818    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:28:35.721000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:35.720977    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"]
Oct 29 22:28:36.587939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.586815    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" event={"ID":"fb326055-bb08-42f0-8597-ddfe942b2ca5","Type":"ContainerStarted","Data":"7544e8a2c87c255d120de64015a9dff6c4ff45e8e041bfcf3c5da2dc82068a1b"}
Oct 29 22:28:36.587939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.586894    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" event={"ID":"fb326055-bb08-42f0-8597-ddfe942b2ca5","Type":"ContainerStarted","Data":"081aa9aaf4b49e9cfc942d5b182331e877f888c78dc4a697cc6092cb034df64d"}
Oct 29 22:28:36.721019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.719434    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" podStartSLOduration=1.719416701 podStartE2EDuration="1.719416701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:36.652739005 +0000 UTC m=+858.677515136" watchObservedRunningTime="2026-10-29 22:28:36.719416701 +0000 UTC m=+858.744192832"
Oct 29 22:28:36.840983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.840924    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:36.916137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.914599    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"]
Oct 29 22:28:36.916137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.915508    4860 cpu_manager.go:414] "containerMap: removing container" podUID="399fff50-fdaf-4acb-9805-ac08d8313079" containerName="revoke-keys"
Oct 29 22:28:36.916137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.915527    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399fff50-fdaf-4acb-9805-ac08d8313079" containerName="revoke-keys"
Oct 29 22:28:36.916137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.915669    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="399fff50-fdaf-4acb-9805-ac08d8313079" containerName="revoke-keys"
Oct 29 22:28:36.931340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.930561    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"]
Oct 29 22:28:36.931556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.931449    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:36.939529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.937718    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-xks-ca\") pod \"399fff50-fdaf-4acb-9805-ac08d8313079\" (UID: \"399fff50-fdaf-4acb-9805-ac08d8313079\") "
Oct 29 22:28:36.939529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.938176    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-openshift-ca\") pod \"399fff50-fdaf-4acb-9805-ac08d8313079\" (UID: \"399fff50-fdaf-4acb-9805-ac08d8313079\") "
Oct 29 22:28:36.939529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.938374    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399fff50-fdaf-4acb-9805-ac08d8313079-kube-api-access-8zqzq\" (UniqueName: \"kubernetes.io/projected/399fff50-fdaf-4acb-9805-ac08d8313079-kube-api-access-8zqzq\") pod \"399fff50-fdaf-4acb-9805-ac08d8313079\" (UID: \"399fff50-fdaf-4acb-9805-ac08d8313079\") "
Oct 29 22:28:36.939751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.939554    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-openshift-ca" pod "399fff50-fdaf-4acb-9805-ac08d8313079" (UID: "399fff50-fdaf-4acb-9805-ac08d8313079"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:28:36.942566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.942338    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-xks-ca" pod "399fff50-fdaf-4acb-9805-ac08d8313079" (UID: "399fff50-fdaf-4acb-9805-ac08d8313079"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:28:36.943373 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.943309    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399fff50-fdaf-4acb-9805-ac08d8313079-kube-api-access-8zqzq" pod "399fff50-fdaf-4acb-9805-ac08d8313079" (UID: "399fff50-fdaf-4acb-9805-ac08d8313079"). InnerVolumeSpecName "kube-api-access-8zqzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:28:36.996953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:36.996787    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:37.039445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.039411    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkpxw\" (UniqueName: \"kubernetes.io/projected/58a3308c-f2a3-457d-8771-16f445e50b63-kube-api-access-rkpxw\") pod \"maas-api-revoke-sub-5a027f8e-w4gtn\" (UID: \"58a3308c-f2a3-457d-8771-16f445e50b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:37.039638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.039505    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a027f8e-w4gtn\" (UID: \"58a3308c-f2a3-457d-8771-16f445e50b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:37.039638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.039613    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a027f8e-w4gtn\" (UID: \"58a3308c-f2a3-457d-8771-16f445e50b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:37.039795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.039778    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zqzq\" (UniqueName: \"kubernetes.io/projected/399fff50-fdaf-4acb-9805-ac08d8313079-kube-api-access-8zqzq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:37.039831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.039799    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:37.039831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.039810    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399fff50-fdaf-4acb-9805-ac08d8313079-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:37.140580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.140468    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a027f8e-w4gtn\" (UID: \"58a3308c-f2a3-457d-8771-16f445e50b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:37.140780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.140677    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkpxw\" (UniqueName: \"kubernetes.io/projected/58a3308c-f2a3-457d-8771-16f445e50b63-kube-api-access-rkpxw\") pod \"maas-api-revoke-sub-5a027f8e-w4gtn\" (UID: \"58a3308c-f2a3-457d-8771-16f445e50b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:37.140780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.140709    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a027f8e-w4gtn\" (UID: \"58a3308c-f2a3-457d-8771-16f445e50b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:37.141657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.141625    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a027f8e-w4gtn\" (UID: \"58a3308c-f2a3-457d-8771-16f445e50b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:37.145016 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.144991    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a027f8e-w4gtn\" (UID: \"58a3308c-f2a3-457d-8771-16f445e50b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:37.163278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.163253    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkpxw\" (UniqueName: \"kubernetes.io/projected/58a3308c-f2a3-457d-8771-16f445e50b63-kube-api-access-rkpxw\") pod \"maas-api-revoke-sub-5a027f8e-w4gtn\" (UID: \"58a3308c-f2a3-457d-8771-16f445e50b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:37.315669 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.315636    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:37.400732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.400641    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:37.404402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.404342    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:28:37.404532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.404413    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" podUID="fb326055-bb08-42f0-8597-ddfe942b2ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 29 22:28:37.599567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.599527    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw" event={"ID":"399fff50-fdaf-4acb-9805-ac08d8313079","Type":"ContainerDied","Data":"89fee656881131f5078da8093b83b255543741971641259a99fab7e2b35d8de8"}
Oct 29 22:28:37.599938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.599585    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89fee656881131f5078da8093b83b255543741971641259a99fab7e2b35d8de8"
Oct 29 22:28:37.599938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.599547    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"
Oct 29 22:28:37.622948 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:28:37.622917    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a3308c_f2a3_457d_8771_16f445e50b63.slice/crio-115dbcb1b5a8a071683dad1cb35b4a0b005667f5cd3ac95d38ce46630c129c90 WatchSource:0}: Error finding container 115dbcb1b5a8a071683dad1cb35b4a0b005667f5cd3ac95d38ce46630c129c90: Status 404 returned error can't find the container with id 115dbcb1b5a8a071683dad1cb35b4a0b005667f5cd3ac95d38ce46630c129c90
Oct 29 22:28:37.624042 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:37.624020    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"]
Oct 29 22:28:38.408027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:38.407873    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:28:38.408027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:38.407945    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" podUID="fb326055-bb08-42f0-8597-ddfe942b2ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 29 22:28:38.612135 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:38.611446    4860 generic.go:356] "Generic (PLEG): container finished" podID="58a3308c-f2a3-457d-8771-16f445e50b63" containerID="5de4998baee2c7d81acf9d8cdf1822c14ed32a2e87c11a5e950e3d317ef07132" exitCode=0
Oct 29 22:28:38.612135 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:38.611526    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn" event={"ID":"58a3308c-f2a3-457d-8771-16f445e50b63","Type":"ContainerDied","Data":"5de4998baee2c7d81acf9d8cdf1822c14ed32a2e87c11a5e950e3d317ef07132"}
Oct 29 22:28:38.612135 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:38.611619    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn" event={"ID":"58a3308c-f2a3-457d-8771-16f445e50b63","Type":"ContainerStarted","Data":"115dbcb1b5a8a071683dad1cb35b4a0b005667f5cd3ac95d38ce46630c129c90"}
Oct 29 22:28:38.692900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:38.692777    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:39.399809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:39.399186    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:28:39.399809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:39.399284    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" podUID="fb326055-bb08-42f0-8597-ddfe942b2ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 29 22:28:39.853603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:39.853578    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:39.901627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:39.901597    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-xks-ca\") pod \"58a3308c-f2a3-457d-8771-16f445e50b63\" (UID: \"58a3308c-f2a3-457d-8771-16f445e50b63\") "
Oct 29 22:28:39.901792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:39.901651    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a3308c-f2a3-457d-8771-16f445e50b63-kube-api-access-rkpxw\" (UniqueName: \"kubernetes.io/projected/58a3308c-f2a3-457d-8771-16f445e50b63-kube-api-access-rkpxw\") pod \"58a3308c-f2a3-457d-8771-16f445e50b63\" (UID: \"58a3308c-f2a3-457d-8771-16f445e50b63\") "
Oct 29 22:28:39.901792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:39.901751    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-openshift-ca\") pod \"58a3308c-f2a3-457d-8771-16f445e50b63\" (UID: \"58a3308c-f2a3-457d-8771-16f445e50b63\") "
Oct 29 22:28:39.902526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:39.902499    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-openshift-ca" pod "58a3308c-f2a3-457d-8771-16f445e50b63" (UID: "58a3308c-f2a3-457d-8771-16f445e50b63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:28:39.905066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:39.905035    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-xks-ca" pod "58a3308c-f2a3-457d-8771-16f445e50b63" (UID: "58a3308c-f2a3-457d-8771-16f445e50b63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:28:39.907816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:39.907698    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a3308c-f2a3-457d-8771-16f445e50b63-kube-api-access-rkpxw" pod "58a3308c-f2a3-457d-8771-16f445e50b63" (UID: "58a3308c-f2a3-457d-8771-16f445e50b63"). InnerVolumeSpecName "kube-api-access-rkpxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:28:40.004260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.004192    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:40.004260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.004217    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkpxw\" (UniqueName: \"kubernetes.io/projected/58a3308c-f2a3-457d-8771-16f445e50b63-kube-api-access-rkpxw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:40.004260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.004227    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a3308c-f2a3-457d-8771-16f445e50b63-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:40.399298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.399215    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:28:40.399298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.399267    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" podUID="fb326055-bb08-42f0-8597-ddfe942b2ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 29 22:28:40.519060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.519018    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:40.537246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.537212    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"]
Oct 29 22:28:40.537766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.537752    4860 cpu_manager.go:414] "containerMap: removing container" podUID="58a3308c-f2a3-457d-8771-16f445e50b63" containerName="revoke-keys"
Oct 29 22:28:40.537820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.537767    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a3308c-f2a3-457d-8771-16f445e50b63" containerName="revoke-keys"
Oct 29 22:28:40.537870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.537860    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a3308c-f2a3-457d-8771-16f445e50b63" containerName="revoke-keys"
Oct 29 22:28:40.541232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.541215    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:40.557868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.557846    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"]
Oct 29 22:28:40.610890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.610860    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a96dc7b9-qx6lv\" (UID: \"c7314749-483a-4958-9bfd-836488a4f14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:40.611178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.611161    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a96dc7b9-qx6lv\" (UID: \"c7314749-483a-4958-9bfd-836488a4f14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:40.611455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.611430    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9skth\" (UniqueName: \"kubernetes.io/projected/c7314749-483a-4958-9bfd-836488a4f14c-kube-api-access-9skth\") pod \"maas-api-revoke-sub-a96dc7b9-qx6lv\" (UID: \"c7314749-483a-4958-9bfd-836488a4f14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:40.627441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.627419    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"
Oct 29 22:28:40.627568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.627460    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn" event={"ID":"58a3308c-f2a3-457d-8771-16f445e50b63","Type":"ContainerDied","Data":"115dbcb1b5a8a071683dad1cb35b4a0b005667f5cd3ac95d38ce46630c129c90"}
Oct 29 22:28:40.627568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.627513    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="115dbcb1b5a8a071683dad1cb35b4a0b005667f5cd3ac95d38ce46630c129c90"
Oct 29 22:28:40.712911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.712877    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9skth\" (UniqueName: \"kubernetes.io/projected/c7314749-483a-4958-9bfd-836488a4f14c-kube-api-access-9skth\") pod \"maas-api-revoke-sub-a96dc7b9-qx6lv\" (UID: \"c7314749-483a-4958-9bfd-836488a4f14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:40.713080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.712920    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a96dc7b9-qx6lv\" (UID: \"c7314749-483a-4958-9bfd-836488a4f14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:40.713080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.712954    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a96dc7b9-qx6lv\" (UID: \"c7314749-483a-4958-9bfd-836488a4f14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:40.713682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.713656    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a96dc7b9-qx6lv\" (UID: \"c7314749-483a-4958-9bfd-836488a4f14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:40.715382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.715361    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a96dc7b9-qx6lv\" (UID: \"c7314749-483a-4958-9bfd-836488a4f14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:40.733825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.733799    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9skth\" (UniqueName: \"kubernetes.io/projected/c7314749-483a-4958-9bfd-836488a4f14c-kube-api-access-9skth\") pod \"maas-api-revoke-sub-a96dc7b9-qx6lv\" (UID: \"c7314749-483a-4958-9bfd-836488a4f14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:40.855143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:40.855111    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:41.215363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:41.214880    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"]
Oct 29 22:28:41.398855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:41.398826    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 29 22:28:41.399015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:41.398876    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" podUID="fb326055-bb08-42f0-8597-ddfe942b2ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 29 22:28:41.635915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:41.635878    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv" event={"ID":"c7314749-483a-4958-9bfd-836488a4f14c","Type":"ContainerStarted","Data":"cacebe49b874d9effd21c97876c307684d9c5bb321639da666d922ec10831b3a"}
Oct 29 22:28:41.636071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:41.635938    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv" event={"ID":"c7314749-483a-4958-9bfd-836488a4f14c","Type":"ContainerStarted","Data":"5ff88eb8965ada3ff5a03a2abff91d5b2028c310d8993cf966a8977388a1d7a3"}
Oct 29 22:28:41.682458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:41.681621    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv" podStartSLOduration=1.681600933 podStartE2EDuration="1.681600933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:41.680634459 +0000 UTC m=+863.705410620" watchObservedRunningTime="2026-10-29 22:28:41.681600933 +0000 UTC m=+863.706377065"
Oct 29 22:28:42.286292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:42.286254    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:42.586465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:42.586380    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:42.586765 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:42.586740    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:42.587566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:42.587549    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:28:42.643833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:42.643803    4860 generic.go:356] "Generic (PLEG): container finished" podID="c7314749-483a-4958-9bfd-836488a4f14c" containerID="cacebe49b874d9effd21c97876c307684d9c5bb321639da666d922ec10831b3a" exitCode=0
Oct 29 22:28:42.643974 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:42.643873    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv" event={"ID":"c7314749-483a-4958-9bfd-836488a4f14c","Type":"ContainerDied","Data":"cacebe49b874d9effd21c97876c307684d9c5bb321639da666d922ec10831b3a"}
Oct 29 22:28:43.823270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:43.823249    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:43.852372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:43.852312    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-openshift-ca\") pod \"c7314749-483a-4958-9bfd-836488a4f14c\" (UID: \"c7314749-483a-4958-9bfd-836488a4f14c\") "
Oct 29 22:28:43.852592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:43.852408    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7314749-483a-4958-9bfd-836488a4f14c-kube-api-access-9skth\" (UniqueName: \"kubernetes.io/projected/c7314749-483a-4958-9bfd-836488a4f14c-kube-api-access-9skth\") pod \"c7314749-483a-4958-9bfd-836488a4f14c\" (UID: \"c7314749-483a-4958-9bfd-836488a4f14c\") "
Oct 29 22:28:43.852592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:43.852507    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-xks-ca\") pod \"c7314749-483a-4958-9bfd-836488a4f14c\" (UID: \"c7314749-483a-4958-9bfd-836488a4f14c\") "
Oct 29 22:28:43.853377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:43.853344    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-openshift-ca" pod "c7314749-483a-4958-9bfd-836488a4f14c" (UID: "c7314749-483a-4958-9bfd-836488a4f14c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:28:43.855161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:43.855106    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-xks-ca" pod "c7314749-483a-4958-9bfd-836488a4f14c" (UID: "c7314749-483a-4958-9bfd-836488a4f14c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:28:43.855708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:43.855684    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7314749-483a-4958-9bfd-836488a4f14c-kube-api-access-9skth" pod "c7314749-483a-4958-9bfd-836488a4f14c" (UID: "c7314749-483a-4958-9bfd-836488a4f14c"). InnerVolumeSpecName "kube-api-access-9skth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:28:43.953674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:43.953640    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:43.953840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:43.953680    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9skth\" (UniqueName: \"kubernetes.io/projected/c7314749-483a-4958-9bfd-836488a4f14c-kube-api-access-9skth\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:43.953840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:43.953695    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7314749-483a-4958-9bfd-836488a4f14c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:43.965883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:43.965859    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:44.288511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.288427    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"]
Oct 29 22:28:44.288980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.288961    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c7314749-483a-4958-9bfd-836488a4f14c" containerName="revoke-keys"
Oct 29 22:28:44.289080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.288982    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7314749-483a-4958-9bfd-836488a4f14c" containerName="revoke-keys"
Oct 29 22:28:44.289170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.289157    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7314749-483a-4958-9bfd-836488a4f14c" containerName="revoke-keys"
Oct 29 22:28:44.292465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.292446    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:44.310257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.310232    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"]
Oct 29 22:28:44.358187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.358157    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ccd153-xkbhf\" (UID: \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:44.358318 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.358299    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ccd153-xkbhf\" (UID: \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:44.358414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.358400    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9vj5\" (UniqueName: \"kubernetes.io/projected/1aa87eb1-b336-4152-a079-ea82d1bcaac4-kube-api-access-s9vj5\") pod \"maas-api-revoke-sub-62ccd153-xkbhf\" (UID: \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:44.459462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.459436    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ccd153-xkbhf\" (UID: \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:44.459570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.459501    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9vj5\" (UniqueName: \"kubernetes.io/projected/1aa87eb1-b336-4152-a079-ea82d1bcaac4-kube-api-access-s9vj5\") pod \"maas-api-revoke-sub-62ccd153-xkbhf\" (UID: \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:44.459570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.459541    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ccd153-xkbhf\" (UID: \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:44.460226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.460208    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ccd153-xkbhf\" (UID: \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:44.461976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.461958    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ccd153-xkbhf\" (UID: \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:44.480218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.480195    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9vj5\" (UniqueName: \"kubernetes.io/projected/1aa87eb1-b336-4152-a079-ea82d1bcaac4-kube-api-access-s9vj5\") pod \"maas-api-revoke-sub-62ccd153-xkbhf\" (UID: \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:44.605687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.605638    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:44.660694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.660664    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"
Oct 29 22:28:44.660782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.660707    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv" event={"ID":"c7314749-483a-4958-9bfd-836488a4f14c","Type":"ContainerDied","Data":"5ff88eb8965ada3ff5a03a2abff91d5b2028c310d8993cf966a8977388a1d7a3"}
Oct 29 22:28:44.660782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.660766    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ff88eb8965ada3ff5a03a2abff91d5b2028c310d8993cf966a8977388a1d7a3"
Oct 29 22:28:44.899753 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:28:44.899723    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa87eb1_b336_4152_a079_ea82d1bcaac4.slice/crio-f5cfd03390999fbcb7ff7db8c7ee39547d7dc9764fb80e6a4bfbcde140d2fbb0 WatchSource:0}: Error finding container f5cfd03390999fbcb7ff7db8c7ee39547d7dc9764fb80e6a4bfbcde140d2fbb0: Status 404 returned error can't find the container with id f5cfd03390999fbcb7ff7db8c7ee39547d7dc9764fb80e6a4bfbcde140d2fbb0
Oct 29 22:28:44.901133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:44.901108    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"]
Oct 29 22:28:45.669399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:45.669368    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf" event={"ID":"1aa87eb1-b336-4152-a079-ea82d1bcaac4","Type":"ContainerStarted","Data":"ecc1c2963d7edbad97c97719b37c787ed2324b57eb48fd0c8a44db0dab1cca91"}
Oct 29 22:28:45.669573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:45.669432    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf" event={"ID":"1aa87eb1-b336-4152-a079-ea82d1bcaac4","Type":"ContainerStarted","Data":"f5cfd03390999fbcb7ff7db8c7ee39547d7dc9764fb80e6a4bfbcde140d2fbb0"}
Oct 29 22:28:45.673112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:45.673078    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:45.711521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:45.711477    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf" podStartSLOduration=1.711464136 podStartE2EDuration="1.711464136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:45.710270579 +0000 UTC m=+867.735046704" watchObservedRunningTime="2026-10-29 22:28:45.711464136 +0000 UTC m=+867.736240268"
Oct 29 22:28:46.677171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:46.677136    4860 generic.go:356] "Generic (PLEG): container finished" podID="1aa87eb1-b336-4152-a079-ea82d1bcaac4" containerID="ecc1c2963d7edbad97c97719b37c787ed2324b57eb48fd0c8a44db0dab1cca91" exitCode=0
Oct 29 22:28:46.677581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:46.677191    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf" event={"ID":"1aa87eb1-b336-4152-a079-ea82d1bcaac4","Type":"ContainerDied","Data":"ecc1c2963d7edbad97c97719b37c787ed2324b57eb48fd0c8a44db0dab1cca91"}
Oct 29 22:28:47.438500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:47.438462    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:47.837218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:47.837193    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:47.895127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:47.895097    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa87eb1-b336-4152-a079-ea82d1bcaac4-kube-api-access-s9vj5\" (UniqueName: \"kubernetes.io/projected/1aa87eb1-b336-4152-a079-ea82d1bcaac4-kube-api-access-s9vj5\") pod \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\" (UID: \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\") "
Oct 29 22:28:47.895318 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:47.895299    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-openshift-ca\") pod \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\" (UID: \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\") "
Oct 29 22:28:47.895422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:47.895387    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-xks-ca\") pod \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\" (UID: \"1aa87eb1-b336-4152-a079-ea82d1bcaac4\") "
Oct 29 22:28:47.895714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:47.895688    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-openshift-ca" pod "1aa87eb1-b336-4152-a079-ea82d1bcaac4" (UID: "1aa87eb1-b336-4152-a079-ea82d1bcaac4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:28:47.895859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:47.895840    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:47.897447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:47.897417    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa87eb1-b336-4152-a079-ea82d1bcaac4-kube-api-access-s9vj5" pod "1aa87eb1-b336-4152-a079-ea82d1bcaac4" (UID: "1aa87eb1-b336-4152-a079-ea82d1bcaac4"). InnerVolumeSpecName "kube-api-access-s9vj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:28:47.897933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:47.897908    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-xks-ca" pod "1aa87eb1-b336-4152-a079-ea82d1bcaac4" (UID: "1aa87eb1-b336-4152-a079-ea82d1bcaac4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:28:47.996982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:47.996919    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa87eb1-b336-4152-a079-ea82d1bcaac4-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:47.996982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:47.996945    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9vj5\" (UniqueName: \"kubernetes.io/projected/1aa87eb1-b336-4152-a079-ea82d1bcaac4-kube-api-access-s9vj5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:48.062895 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:28:48.062860    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:28:48.691062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:48.691034    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"
Oct 29 22:28:48.691223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:48.691069    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf" event={"ID":"1aa87eb1-b336-4152-a079-ea82d1bcaac4","Type":"ContainerDied","Data":"f5cfd03390999fbcb7ff7db8c7ee39547d7dc9764fb80e6a4bfbcde140d2fbb0"}
Oct 29 22:28:48.691223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:48.691123    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5cfd03390999fbcb7ff7db8c7ee39547d7dc9764fb80e6a4bfbcde140d2fbb0"
Oct 29 22:28:49.079435 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:49.077117    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:50.224943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.224906    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"]
Oct 29 22:28:50.225722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.225699    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1aa87eb1-b336-4152-a079-ea82d1bcaac4" containerName="revoke-keys"
Oct 29 22:28:50.225824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.225723    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa87eb1-b336-4152-a079-ea82d1bcaac4" containerName="revoke-keys"
Oct 29 22:28:50.225964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.225950    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa87eb1-b336-4152-a079-ea82d1bcaac4" containerName="revoke-keys"
Oct 29 22:28:50.230457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.230437    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.264787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.264762    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-22d634-openshift-default-dockercfg-8sxjc\""
Oct 29 22:28:50.270580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.270554    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"]
Oct 29 22:28:50.320359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.320312    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.320529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.320440    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.320529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.320494    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.320529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.320523    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.320709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.320538    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnm8c\" (UniqueName: \"kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-kube-api-access-cnm8c\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.320709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.320599    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13f5a1b-9637-4492-81c3-874e6c91294d-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.320709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.320670    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.320824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.320766    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.320824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.320794    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.421987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.421954    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.422124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.421999    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.422124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.422056    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.422211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.422186    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.422259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.422235    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.422334 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.422276    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.422398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.422379    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnm8c\" (UniqueName: \"kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-kube-api-access-cnm8c\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.422450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.422415    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.422450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.422425    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13f5a1b-9637-4492-81c3-874e6c91294d-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.422550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.422518    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.422894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.422877    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.422995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.422906    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.423043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.423024    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.423113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.423095    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13f5a1b-9637-4492-81c3-874e6c91294d-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.424589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.424568    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.424732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.424716    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.443144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.443118    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.443471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.443453    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnm8c\" (UniqueName: \"kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-kube-api-access-cnm8c\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.506280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.506192    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"]
Oct 29 22:28:50.512343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.512308    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.530754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.530728    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2ba10a-openshift-default-dockercfg-n9ldm\""
Oct 29 22:28:50.532007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.531985    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"]
Oct 29 22:28:50.546750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.546729    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:50.624171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.624131    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrrb\" (UniqueName: \"kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-kube-api-access-8vrrb\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.624286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.624234    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.624365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.624294    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.624365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.624341    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.624506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.624406    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.624506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.624464    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.624614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.624507    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cc6760-f3f7-45bd-9c31-655c607c4505-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.624614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.624602    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.624715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.624648    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.705287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.705258    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"]
Oct 29 22:28:50.710896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.710876    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.726003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.725983    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726017    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726049    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrrb\" (UniqueName: \"kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-kube-api-access-8vrrb\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726095    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726140    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726171    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726224    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726276    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726317    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cc6760-f3f7-45bd-9c31-655c607c4505-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726490    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726519    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726648    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726734    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.726971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.726954    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cc6760-f3f7-45bd-9c31-655c607c4505-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.728466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.728441    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.728558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.728495    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.734924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.734905    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"]
Oct 29 22:28:50.742959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.742923    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2e046a-openshift-default-dockercfg-5g58t\""
Oct 29 22:28:50.751687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.751663    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.751861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.751838    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrrb\" (UniqueName: \"kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-kube-api-access-8vrrb\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.821963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.821900    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"]
Oct 29 22:28:50.826400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.826374    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:50.827494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.827479    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:50.827807 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.827784    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbkcw\" (UniqueName: \"kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-kube-api-access-fbkcw\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.827872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.827840    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.827930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.827871    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.827993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.827977    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.828029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.828014    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.828079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.828045    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.828141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.828130    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.828183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.828152    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.828183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.828170    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.843604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.843577    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:28:50.843692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.843621    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:28:50.843692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.843620    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:28:50.843794 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.843745    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:28:50.847763 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.844599    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"]
Oct 29 22:28:50.885305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.885271    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"]
Oct 29 22:28:50.885511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.885313    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:50.895531 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:28:50.895495    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13f5a1b_9637_4492_81c3_874e6c91294d.slice/crio-4cf8902949ad318eace7d10f03cdc4b19222e2a49c37e1d7a5ce071bc997fd3b WatchSource:0}: Error finding container 4cf8902949ad318eace7d10f03cdc4b19222e2a49c37e1d7a5ce071bc997fd3b: Status 404 returned error can't find the container with id 4cf8902949ad318eace7d10f03cdc4b19222e2a49c37e1d7a5ce071bc997fd3b
Oct 29 22:28:50.899991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.899960    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:28:50.900136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.900122    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:28:50.900222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.900211    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.929939    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8szv\" (UniqueName: \"kubernetes.io/projected/330b7223-95fc-412f-8825-547da318f55f-kube-api-access-d8szv\") pod \"maas-api-revoke-sub-8d8ffb13-j7n8x\" (UID: \"330b7223-95fc-412f-8825-547da318f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930010    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930053    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930098    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930124    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930141    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930158    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330b7223-95fc-412f-8825-547da318f55f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d8ffb13-j7n8x\" (UID: \"330b7223-95fc-412f-8825-547da318f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930207    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbkcw\" (UniqueName: \"kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-kube-api-access-fbkcw\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930263    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930306    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930377    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930399    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330b7223-95fc-412f-8825-547da318f55f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d8ffb13-j7n8x\" (UID: \"330b7223-95fc-412f-8825-547da318f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930726    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.930909    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.931095    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.931885    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.933351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.932576    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.937350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.934928    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.940911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.940565    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.954369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.953870    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:50.963363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:50.963338    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbkcw\" (UniqueName: \"kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-kube-api-access-fbkcw\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:51.025468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.025179    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:51.047346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.040135    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330b7223-95fc-412f-8825-547da318f55f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d8ffb13-j7n8x\" (UID: \"330b7223-95fc-412f-8825-547da318f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:51.047346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.040187    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8szv\" (UniqueName: \"kubernetes.io/projected/330b7223-95fc-412f-8825-547da318f55f-kube-api-access-d8szv\") pod \"maas-api-revoke-sub-8d8ffb13-j7n8x\" (UID: \"330b7223-95fc-412f-8825-547da318f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:51.047346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.040263    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330b7223-95fc-412f-8825-547da318f55f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d8ffb13-j7n8x\" (UID: \"330b7223-95fc-412f-8825-547da318f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:51.049147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.048549    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330b7223-95fc-412f-8825-547da318f55f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d8ffb13-j7n8x\" (UID: \"330b7223-95fc-412f-8825-547da318f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:51.075398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.073694    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330b7223-95fc-412f-8825-547da318f55f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d8ffb13-j7n8x\" (UID: \"330b7223-95fc-412f-8825-547da318f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:51.078118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.078057    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8szv\" (UniqueName: \"kubernetes.io/projected/330b7223-95fc-412f-8825-547da318f55f-kube-api-access-d8szv\") pod \"maas-api-revoke-sub-8d8ffb13-j7n8x\" (UID: \"330b7223-95fc-412f-8825-547da318f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:51.157346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.149102    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:51.190408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.188781    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"]
Oct 29 22:28:51.192515 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:28:51.192474    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cc6760_f3f7_45bd_9c31_655c607c4505.slice/crio-0a7de8e601e4ea4adbba2881d0551b15ed14096b1577f90128a20eaba87f8faf WatchSource:0}: Error finding container 0a7de8e601e4ea4adbba2881d0551b15ed14096b1577f90128a20eaba87f8faf: Status 404 returned error can't find the container with id 0a7de8e601e4ea4adbba2881d0551b15ed14096b1577f90128a20eaba87f8faf
Oct 29 22:28:51.196405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.196194    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:28:51.196405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.196279    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:28:51.197680 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.196580    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:28:51.232675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.232646    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"]
Oct 29 22:28:51.309916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.309888    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"]
Oct 29 22:28:51.310174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.310159    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:51.373731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.373113    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf2q4\" (UniqueName: \"kubernetes.io/projected/78a60753-01c7-4486-b99f-6d3f2178f07e-kube-api-access-jf2q4\") pod \"maas-api-revoke-sub-65c7a8ca-zgnn7\" (UID: \"78a60753-01c7-4486-b99f-6d3f2178f07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:51.374413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.374001    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c7a8ca-zgnn7\" (UID: \"78a60753-01c7-4486-b99f-6d3f2178f07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:51.374413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.374187    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c7a8ca-zgnn7\" (UID: \"78a60753-01c7-4486-b99f-6d3f2178f07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:51.477517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.475675    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf2q4\" (UniqueName: \"kubernetes.io/projected/78a60753-01c7-4486-b99f-6d3f2178f07e-kube-api-access-jf2q4\") pod \"maas-api-revoke-sub-65c7a8ca-zgnn7\" (UID: \"78a60753-01c7-4486-b99f-6d3f2178f07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:51.477517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.475746    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c7a8ca-zgnn7\" (UID: \"78a60753-01c7-4486-b99f-6d3f2178f07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:51.477517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.475904    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c7a8ca-zgnn7\" (UID: \"78a60753-01c7-4486-b99f-6d3f2178f07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:51.477517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.477437    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c7a8ca-zgnn7\" (UID: \"78a60753-01c7-4486-b99f-6d3f2178f07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:51.492277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.492248    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"]
Oct 29 22:28:51.493505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.493441    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c7a8ca-zgnn7\" (UID: \"78a60753-01c7-4486-b99f-6d3f2178f07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:51.506601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.506576    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf2q4\" (UniqueName: \"kubernetes.io/projected/78a60753-01c7-4486-b99f-6d3f2178f07e-kube-api-access-jf2q4\") pod \"maas-api-revoke-sub-65c7a8ca-zgnn7\" (UID: \"78a60753-01c7-4486-b99f-6d3f2178f07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:51.600718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.594540    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:28:51.600718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.594627    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:28:51.600718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.594670    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:28:51.660849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.653526    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:51.785380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.782536    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"]
Oct 29 22:28:51.940633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.906667    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" event={"ID":"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5","Type":"ContainerStarted","Data":"2ff851f4d624ed0b626e99f0e8fdf62df3acafcf07947eeaff40dddb041d1aa0"}
Oct 29 22:28:52.015379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.997354    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt" event={"ID":"14cc6760-f3f7-45bd-9c31-655c607c4505","Type":"ContainerStarted","Data":"f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0"}
Oct 29 22:28:52.015379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:51.997421    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt" event={"ID":"14cc6760-f3f7-45bd-9c31-655c607c4505","Type":"ContainerStarted","Data":"0a7de8e601e4ea4adbba2881d0551b15ed14096b1577f90128a20eaba87f8faf"}
Oct 29 22:28:52.303772 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:52.285503    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn" podStartSLOduration=2.285483168 podStartE2EDuration="2.285483168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:52.28497929 +0000 UTC m=+874.309755440" watchObservedRunningTime="2026-10-29 22:28:52.285483168 +0000 UTC m=+874.310259305"
Oct 29 22:28:52.346258 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:52.346191    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt" podStartSLOduration=2.346172373 podStartE2EDuration="2.346172373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:52.095353953 +0000 UTC m=+874.120130086" watchObservedRunningTime="2026-10-29 22:28:52.346172373 +0000 UTC m=+874.370948506"
Oct 29 22:28:52.418164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:52.418107    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn" event={"ID":"d13f5a1b-9637-4492-81c3-874e6c91294d","Type":"ContainerStarted","Data":"0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83"}
Oct 29 22:28:52.423518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:52.423484    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn" event={"ID":"d13f5a1b-9637-4492-81c3-874e6c91294d","Type":"ContainerStarted","Data":"4cf8902949ad318eace7d10f03cdc4b19222e2a49c37e1d7a5ce071bc997fd3b"}
Oct 29 22:28:52.566409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:52.554984    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:52.566409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:52.556256    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:28:52.566409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:52.556318    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn" podUID="d13f5a1b-9637-4492-81c3-874e6c91294d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 29 22:28:52.655177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:52.655055    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:52.753117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:52.753088    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"]
Oct 29 22:28:52.829746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:52.829225    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:52.830304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:52.829953    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 22:28:52.830304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:52.830014    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt" podUID="14cc6760-f3f7-45bd-9c31-655c607c4505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 22:28:53.284589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:53.284547    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x" event={"ID":"330b7223-95fc-412f-8825-547da318f55f","Type":"ContainerStarted","Data":"d2589eb597c5c149d6698544a0113776ae5a470395d25bec6a0a0611c27b9c59"}
Oct 29 22:28:53.288434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:53.288398    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x" event={"ID":"330b7223-95fc-412f-8825-547da318f55f","Type":"ContainerStarted","Data":"cb51d7d3ac09d82f4acc6fcbcf2dd537ef03be6997a446b0863dbe0f566b3890"}
Oct 29 22:28:53.342352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:53.341575    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" event={"ID":"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5","Type":"ContainerStarted","Data":"7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731"}
Oct 29 22:28:53.349067 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:53.348441    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7" event={"ID":"78a60753-01c7-4486-b99f-6d3f2178f07e","Type":"ContainerStarted","Data":"e9ed395d8756956f938727a85c380d5e3fa78aad983adbbe0ac670772a0b7695"}
Oct 29 22:28:53.349745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:53.349700    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x" podStartSLOduration=3.34968678 podStartE2EDuration="3.34968678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:53.348133408 +0000 UTC m=+875.372909563" watchObservedRunningTime="2026-10-29 22:28:53.34968678 +0000 UTC m=+875.374462911"
Oct 29 22:28:53.410526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:53.410448    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7" podStartSLOduration=2.41042749 podStartE2EDuration="2.41042749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:53.409848667 +0000 UTC m=+875.434624799" watchObservedRunningTime="2026-10-29 22:28:53.41042749 +0000 UTC m=+875.435203620"
Oct 29 22:28:53.524416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:53.523380    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" podStartSLOduration=3.523354193 podStartE2EDuration="3.523354193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:53.468737296 +0000 UTC m=+875.493513474" watchObservedRunningTime="2026-10-29 22:28:53.523354193 +0000 UTC m=+875.548130324"
Oct 29 22:28:53.547685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:53.547640    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:28:53.547882 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:53.547700    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn" podUID="d13f5a1b-9637-4492-81c3-874e6c91294d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 29 22:28:53.827924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:53.827822    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 22:28:53.828276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:53.828160    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt" podUID="14cc6760-f3f7-45bd-9c31-655c607c4505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 22:28:54.028623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:54.028566    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:28:54.028927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:54.028885    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" podUID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 29 22:28:54.029374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:54.029357    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:28:54.360643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:54.360606    4860 generic.go:356] "Generic (PLEG): container finished" podID="78a60753-01c7-4486-b99f-6d3f2178f07e" containerID="755f6f211f57fc67e8e720cbbf250a792157736bbf6fdfab103b66dfac5fde42" exitCode=0
Oct 29 22:28:54.361033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:54.360652    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7" event={"ID":"78a60753-01c7-4486-b99f-6d3f2178f07e","Type":"ContainerDied","Data":"755f6f211f57fc67e8e720cbbf250a792157736bbf6fdfab103b66dfac5fde42"}
Oct 29 22:28:54.362342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:54.362303    4860 generic.go:356] "Generic (PLEG): container finished" podID="330b7223-95fc-412f-8825-547da318f55f" containerID="d2589eb597c5c149d6698544a0113776ae5a470395d25bec6a0a0611c27b9c59" exitCode=0
Oct 29 22:28:54.362439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:54.362353    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x" event={"ID":"330b7223-95fc-412f-8825-547da318f55f","Type":"ContainerDied","Data":"d2589eb597c5c149d6698544a0113776ae5a470395d25bec6a0a0611c27b9c59"}
Oct 29 22:28:54.550350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:54.549937    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:28:54.550350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:54.550004    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn" podUID="d13f5a1b-9637-4492-81c3-874e6c91294d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 29 22:28:54.595453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:54.579807    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:54.832346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:54.830726    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 22:28:54.832346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:54.830791    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt" podUID="14cc6760-f3f7-45bd-9c31-655c607c4505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 22:28:55.037349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:55.036816    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:28:55.037349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:55.036873    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" podUID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 29 22:28:55.549350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:55.547991    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:28:55.549350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:55.548054    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn" podUID="d13f5a1b-9637-4492-81c3-874e6c91294d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 29 22:28:55.850025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:55.849991    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 22:28:55.850283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:55.850256    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt" podUID="14cc6760-f3f7-45bd-9c31-655c607c4505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 22:28:56.048555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.042596    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:28:56.048555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.042651    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" podUID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 29 22:28:56.472112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.472076    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"]
Oct 29 22:28:56.544024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.543273    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:56.550507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.549520    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:56.550507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.549971    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:28:56.550507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.550022    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn" podUID="d13f5a1b-9637-4492-81c3-874e6c91294d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 29 22:28:56.557352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.552631    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"]
Oct 29 22:28:56.557352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.552769    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:28:56.557352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.556988    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:56.577348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.567443    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-xks-ca\") pod \"78a60753-01c7-4486-b99f-6d3f2178f07e\" (UID: \"78a60753-01c7-4486-b99f-6d3f2178f07e\") "
Oct 29 22:28:56.577348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.567533    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330b7223-95fc-412f-8825-547da318f55f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330b7223-95fc-412f-8825-547da318f55f-maas-api-openshift-ca\") pod \"330b7223-95fc-412f-8825-547da318f55f\" (UID: \"330b7223-95fc-412f-8825-547da318f55f\") "
Oct 29 22:28:56.577348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.567667    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330b7223-95fc-412f-8825-547da318f55f-kube-api-access-d8szv\" (UniqueName: \"kubernetes.io/projected/330b7223-95fc-412f-8825-547da318f55f-kube-api-access-d8szv\") pod \"330b7223-95fc-412f-8825-547da318f55f\" (UID: \"330b7223-95fc-412f-8825-547da318f55f\") "
Oct 29 22:28:56.577348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.567718    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-openshift-ca\") pod \"78a60753-01c7-4486-b99f-6d3f2178f07e\" (UID: \"78a60753-01c7-4486-b99f-6d3f2178f07e\") "
Oct 29 22:28:56.577348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.567805    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/330b7223-95fc-412f-8825-547da318f55f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330b7223-95fc-412f-8825-547da318f55f-maas-api-xks-ca\") pod \"330b7223-95fc-412f-8825-547da318f55f\" (UID: \"330b7223-95fc-412f-8825-547da318f55f\") "
Oct 29 22:28:56.577348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.567839    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a60753-01c7-4486-b99f-6d3f2178f07e-kube-api-access-jf2q4\" (UniqueName: \"kubernetes.io/projected/78a60753-01c7-4486-b99f-6d3f2178f07e-kube-api-access-jf2q4\") pod \"78a60753-01c7-4486-b99f-6d3f2178f07e\" (UID: \"78a60753-01c7-4486-b99f-6d3f2178f07e\") "
Oct 29 22:28:56.577348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.568249    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157cfc60-csphq\" (UID: \"dfc7d10b-c324-4222-b766-d88a2548f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:28:56.585790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.585760    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6krt\" (UniqueName: \"kubernetes.io/projected/dfc7d10b-c324-4222-b766-d88a2548f84c-kube-api-access-b6krt\") pod \"maas-api-revoke-sub-157cfc60-csphq\" (UID: \"dfc7d10b-c324-4222-b766-d88a2548f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:28:56.585966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.585949    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157cfc60-csphq\" (UID: \"dfc7d10b-c324-4222-b766-d88a2548f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:28:56.586757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.586716    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330b7223-95fc-412f-8825-547da318f55f-maas-api-openshift-ca" pod "330b7223-95fc-412f-8825-547da318f55f" (UID: "330b7223-95fc-412f-8825-547da318f55f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:28:56.596346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.586990    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-openshift-ca" pod "78a60753-01c7-4486-b99f-6d3f2178f07e" (UID: "78a60753-01c7-4486-b99f-6d3f2178f07e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:28:56.596517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.593105    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7" event={"ID":"78a60753-01c7-4486-b99f-6d3f2178f07e","Type":"ContainerDied","Data":"e9ed395d8756956f938727a85c380d5e3fa78aad983adbbe0ac670772a0b7695"}
Oct 29 22:28:56.596926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.596909    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9ed395d8756956f938727a85c380d5e3fa78aad983adbbe0ac670772a0b7695"
Oct 29 22:28:56.597048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.593384    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"
Oct 29 22:28:56.620349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.617853    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-xks-ca" pod "78a60753-01c7-4486-b99f-6d3f2178f07e" (UID: "78a60753-01c7-4486-b99f-6d3f2178f07e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:28:56.664180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.664134    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330b7223-95fc-412f-8825-547da318f55f-kube-api-access-d8szv" pod "330b7223-95fc-412f-8825-547da318f55f" (UID: "330b7223-95fc-412f-8825-547da318f55f"). InnerVolumeSpecName "kube-api-access-d8szv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:28:56.665790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.665458    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x" event={"ID":"330b7223-95fc-412f-8825-547da318f55f","Type":"ContainerDied","Data":"cb51d7d3ac09d82f4acc6fcbcf2dd537ef03be6997a446b0863dbe0f566b3890"}
Oct 29 22:28:56.666031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.665987    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb51d7d3ac09d82f4acc6fcbcf2dd537ef03be6997a446b0863dbe0f566b3890"
Oct 29 22:28:56.666210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.666196    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"
Oct 29 22:28:56.752437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.687999    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6krt\" (UniqueName: \"kubernetes.io/projected/dfc7d10b-c324-4222-b766-d88a2548f84c-kube-api-access-b6krt\") pod \"maas-api-revoke-sub-157cfc60-csphq\" (UID: \"dfc7d10b-c324-4222-b766-d88a2548f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:28:56.752437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.688045    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157cfc60-csphq\" (UID: \"dfc7d10b-c324-4222-b766-d88a2548f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:28:56.752437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.688201    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157cfc60-csphq\" (UID: \"dfc7d10b-c324-4222-b766-d88a2548f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:28:56.752437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.688393    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:56.752437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.688415    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a60753-01c7-4486-b99f-6d3f2178f07e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:56.752437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.688430    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330b7223-95fc-412f-8825-547da318f55f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:56.752437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.688444    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8szv\" (UniqueName: \"kubernetes.io/projected/330b7223-95fc-412f-8825-547da318f55f-kube-api-access-d8szv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:56.752437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.690431    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/330b7223-95fc-412f-8825-547da318f55f-maas-api-xks-ca" pod "330b7223-95fc-412f-8825-547da318f55f" (UID: "330b7223-95fc-412f-8825-547da318f55f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:28:56.752437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.691475    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157cfc60-csphq\" (UID: \"dfc7d10b-c324-4222-b766-d88a2548f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:28:56.752437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.712081    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a60753-01c7-4486-b99f-6d3f2178f07e-kube-api-access-jf2q4" pod "78a60753-01c7-4486-b99f-6d3f2178f07e" (UID: "78a60753-01c7-4486-b99f-6d3f2178f07e"). InnerVolumeSpecName "kube-api-access-jf2q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:28:56.758013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.757974    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157cfc60-csphq\" (UID: \"dfc7d10b-c324-4222-b766-d88a2548f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:28:56.812397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.812126    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6krt\" (UniqueName: \"kubernetes.io/projected/dfc7d10b-c324-4222-b766-d88a2548f84c-kube-api-access-b6krt\") pod \"maas-api-revoke-sub-157cfc60-csphq\" (UID: \"dfc7d10b-c324-4222-b766-d88a2548f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:28:56.836348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.816552    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330b7223-95fc-412f-8825-547da318f55f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:56.836348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.816583    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf2q4\" (UniqueName: \"kubernetes.io/projected/78a60753-01c7-4486-b99f-6d3f2178f07e-kube-api-access-jf2q4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:28:56.836348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.827731    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 22:28:56.836348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.827782    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt" podUID="14cc6760-f3f7-45bd-9c31-655c607c4505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 22:28:56.965652 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:56.965624    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:28:57.050303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:57.050177    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:28:57.050669 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:57.050272    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" podUID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 29 22:28:57.547506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:57.547470    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 29 22:28:57.547678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:57.547534    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn" podUID="d13f5a1b-9637-4492-81c3-874e6c91294d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 29 22:28:57.556174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:57.556147    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"]
Oct 29 22:28:57.557991 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:28:57.557964    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc7d10b_c324_4222_b766_d88a2548f84c.slice/crio-778f50cecb4992388e071be1e464e7df57f9576f5b4b0bfdd8e949c9d09413ad WatchSource:0}: Error finding container 778f50cecb4992388e071be1e464e7df57f9576f5b4b0bfdd8e949c9d09413ad: Status 404 returned error can't find the container with id 778f50cecb4992388e071be1e464e7df57f9576f5b4b0bfdd8e949c9d09413ad
Oct 29 22:28:57.727760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:57.727721    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq" event={"ID":"dfc7d10b-c324-4222-b766-d88a2548f84c","Type":"ContainerStarted","Data":"778f50cecb4992388e071be1e464e7df57f9576f5b4b0bfdd8e949c9d09413ad"}
Oct 29 22:28:57.826932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:57.826892    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 22:28:57.827153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:57.826982    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt" podUID="14cc6760-f3f7-45bd-9c31-655c607c4505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 22:28:58.026908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:58.026870    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:28:58.027093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:58.026936    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" podUID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 29 22:28:58.378354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:58.378186    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:28:58.551476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:58.551448    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:58.551847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:58.551826    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:58.552768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:58.552748    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 22:28:58.736347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:58.735942    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq" event={"ID":"dfc7d10b-c324-4222-b766-d88a2548f84c","Type":"ContainerStarted","Data":"b073e6640a70066345a86fcb60cbdb5b1be3bad513061257dafdef118b3dcfc9"}
Oct 29 22:28:58.782722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:58.782653    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq" podStartSLOduration=3.782635593 podStartE2EDuration="3.782635593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:58.776142047 +0000 UTC m=+880.800918214" watchObservedRunningTime="2026-10-29 22:28:58.782635593 +0000 UTC m=+880.807411724"
Oct 29 22:28:58.931052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:58.931024    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:58.932072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:58.932053    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:58.933060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:58.933033    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:28:59.026364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:59.026256    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:28:59.026364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:59.026304    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" podUID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 29 22:28:59.752710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:59.752516    4860 generic.go:356] "Generic (PLEG): container finished" podID="dfc7d10b-c324-4222-b766-d88a2548f84c" containerID="b073e6640a70066345a86fcb60cbdb5b1be3bad513061257dafdef118b3dcfc9" exitCode=0
Oct 29 22:28:59.753119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:28:59.752670    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq" event={"ID":"dfc7d10b-c324-4222-b766-d88a2548f84c","Type":"ContainerDied","Data":"b073e6640a70066345a86fcb60cbdb5b1be3bad513061257dafdef118b3dcfc9"}
Oct 29 22:29:00.030862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:00.030786    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:29:00.031228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:00.031211    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:29:00.032001 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:00.031983    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 22:29:00.064937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:00.064909    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:00.912396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:00.912369    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:29:00.967249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:00.967222    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc7d10b-c324-4222-b766-d88a2548f84c-kube-api-access-b6krt\" (UniqueName: \"kubernetes.io/projected/dfc7d10b-c324-4222-b766-d88a2548f84c-kube-api-access-b6krt\") pod \"dfc7d10b-c324-4222-b766-d88a2548f84c\" (UID: \"dfc7d10b-c324-4222-b766-d88a2548f84c\") "
Oct 29 22:29:00.967433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:00.967264    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-xks-ca\") pod \"dfc7d10b-c324-4222-b766-d88a2548f84c\" (UID: \"dfc7d10b-c324-4222-b766-d88a2548f84c\") "
Oct 29 22:29:00.967510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:00.967462    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-openshift-ca\") pod \"dfc7d10b-c324-4222-b766-d88a2548f84c\" (UID: \"dfc7d10b-c324-4222-b766-d88a2548f84c\") "
Oct 29 22:29:00.967851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:00.967823    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-openshift-ca" pod "dfc7d10b-c324-4222-b766-d88a2548f84c" (UID: "dfc7d10b-c324-4222-b766-d88a2548f84c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:29:00.967972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:00.967901    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:00.969505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:00.969457    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc7d10b-c324-4222-b766-d88a2548f84c-kube-api-access-b6krt" pod "dfc7d10b-c324-4222-b766-d88a2548f84c" (UID: "dfc7d10b-c324-4222-b766-d88a2548f84c"). InnerVolumeSpecName "kube-api-access-b6krt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:29:00.969606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:00.969547    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-xks-ca" pod "dfc7d10b-c324-4222-b766-d88a2548f84c" (UID: "dfc7d10b-c324-4222-b766-d88a2548f84c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:29:01.069139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.069063    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6krt\" (UniqueName: \"kubernetes.io/projected/dfc7d10b-c324-4222-b766-d88a2548f84c-kube-api-access-b6krt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:01.069139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.069100    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc7d10b-c324-4222-b766-d88a2548f84c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:01.209152 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.209119    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"]
Oct 29 22:29:01.209667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.209654    4860 cpu_manager.go:414] "containerMap: removing container" podUID="dfc7d10b-c324-4222-b766-d88a2548f84c" containerName="revoke-keys"
Oct 29 22:29:01.209716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.209669    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc7d10b-c324-4222-b766-d88a2548f84c" containerName="revoke-keys"
Oct 29 22:29:01.209716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.209686    4860 cpu_manager.go:414] "containerMap: removing container" podUID="78a60753-01c7-4486-b99f-6d3f2178f07e" containerName="revoke-keys"
Oct 29 22:29:01.209716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.209693    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a60753-01c7-4486-b99f-6d3f2178f07e" containerName="revoke-keys"
Oct 29 22:29:01.209811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.209725    4860 cpu_manager.go:414] "containerMap: removing container" podUID="330b7223-95fc-412f-8825-547da318f55f" containerName="revoke-keys"
Oct 29 22:29:01.209811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.209731    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330b7223-95fc-412f-8825-547da318f55f" containerName="revoke-keys"
Oct 29 22:29:01.209872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.209817    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="330b7223-95fc-412f-8825-547da318f55f" containerName="revoke-keys"
Oct 29 22:29:01.209872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.209828    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc7d10b-c324-4222-b766-d88a2548f84c" containerName="revoke-keys"
Oct 29 22:29:01.209872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.209836    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a60753-01c7-4486-b99f-6d3f2178f07e" containerName="revoke-keys"
Oct 29 22:29:01.213562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.213546    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:01.231304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.231274    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"]
Oct 29 22:29:01.271670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.271636    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0eb50da-r7fp6\" (UID: \"0debd5d9-d5f0-4c81-b35f-696613811010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:01.271912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.271890    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6qx7\" (UniqueName: \"kubernetes.io/projected/0debd5d9-d5f0-4c81-b35f-696613811010-kube-api-access-h6qx7\") pod \"maas-api-revoke-sub-c0eb50da-r7fp6\" (UID: \"0debd5d9-d5f0-4c81-b35f-696613811010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:01.272007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.271933    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0eb50da-r7fp6\" (UID: \"0debd5d9-d5f0-4c81-b35f-696613811010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:01.347709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.347634    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"]
Oct 29 22:29:01.351989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.351970    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:01.370601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.370576    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"]
Oct 29 22:29:01.373414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.373389    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6qx7\" (UniqueName: \"kubernetes.io/projected/0debd5d9-d5f0-4c81-b35f-696613811010-kube-api-access-h6qx7\") pod \"maas-api-revoke-sub-c0eb50da-r7fp6\" (UID: \"0debd5d9-d5f0-4c81-b35f-696613811010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:01.373521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.373437    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0eb50da-r7fp6\" (UID: \"0debd5d9-d5f0-4c81-b35f-696613811010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:01.373521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.373516    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0eb50da-r7fp6\" (UID: \"0debd5d9-d5f0-4c81-b35f-696613811010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:01.374680 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.374656    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0eb50da-r7fp6\" (UID: \"0debd5d9-d5f0-4c81-b35f-696613811010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:01.376166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.376144    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0eb50da-r7fp6\" (UID: \"0debd5d9-d5f0-4c81-b35f-696613811010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:01.397976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.397955    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6qx7\" (UniqueName: \"kubernetes.io/projected/0debd5d9-d5f0-4c81-b35f-696613811010-kube-api-access-h6qx7\") pod \"maas-api-revoke-sub-c0eb50da-r7fp6\" (UID: \"0debd5d9-d5f0-4c81-b35f-696613811010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:01.474306 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.474277    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae35e6a-728l8\" (UID: \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:01.474474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.474419    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae35e6a-728l8\" (UID: \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:01.474474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.474462    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmxgk\" (UniqueName: \"kubernetes.io/projected/729aaaa7-6766-4d22-84c0-88bc54b2b74a-kube-api-access-rmxgk\") pod \"maas-api-revoke-sub-dae35e6a-728l8\" (UID: \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:01.527391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.527365    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:01.575378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.575346    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae35e6a-728l8\" (UID: \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:01.575560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.575459    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae35e6a-728l8\" (UID: \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:01.575560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.575528    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmxgk\" (UniqueName: \"kubernetes.io/projected/729aaaa7-6766-4d22-84c0-88bc54b2b74a-kube-api-access-rmxgk\") pod \"maas-api-revoke-sub-dae35e6a-728l8\" (UID: \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:01.576554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.576525    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae35e6a-728l8\" (UID: \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:01.578122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.578101    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae35e6a-728l8\" (UID: \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:01.598451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.598395    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmxgk\" (UniqueName: \"kubernetes.io/projected/729aaaa7-6766-4d22-84c0-88bc54b2b74a-kube-api-access-rmxgk\") pod \"maas-api-revoke-sub-dae35e6a-728l8\" (UID: \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:01.665664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.665636    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:01.768146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.768115    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"
Oct 29 22:29:01.768146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.768131    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq" event={"ID":"dfc7d10b-c324-4222-b766-d88a2548f84c","Type":"ContainerDied","Data":"778f50cecb4992388e071be1e464e7df57f9576f5b4b0bfdd8e949c9d09413ad"}
Oct 29 22:29:01.768422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.768226    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="778f50cecb4992388e071be1e464e7df57f9576f5b4b0bfdd8e949c9d09413ad"
Oct 29 22:29:01.849787 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:29:01.849723    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0debd5d9_d5f0_4c81_b35f_696613811010.slice/crio-0e0753029c50ec354e7e97039a6888debad3a3c1ded76c0859ec261695f049bb WatchSource:0}: Error finding container 0e0753029c50ec354e7e97039a6888debad3a3c1ded76c0859ec261695f049bb: Status 404 returned error can't find the container with id 0e0753029c50ec354e7e97039a6888debad3a3c1ded76c0859ec261695f049bb
Oct 29 22:29:01.851763 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.851740    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"]
Oct 29 22:29:01.895246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.895194    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:01.968978 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:29:01.968905    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729aaaa7_6766_4d22_84c0_88bc54b2b74a.slice/crio-1eb4520b1a34a3260fab96797f40df7e63047d90afe4ffb05629cf8854c98d7b WatchSource:0}: Error finding container 1eb4520b1a34a3260fab96797f40df7e63047d90afe4ffb05629cf8854c98d7b: Status 404 returned error can't find the container with id 1eb4520b1a34a3260fab96797f40df7e63047d90afe4ffb05629cf8854c98d7b
Oct 29 22:29:01.971333 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:01.971293    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"]
Oct 29 22:29:02.779400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:02.779359    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8" event={"ID":"729aaaa7-6766-4d22-84c0-88bc54b2b74a","Type":"ContainerStarted","Data":"4ae528fb19238c30386bac2d5d42808d26f87c0c97bf7faa9ab91932c42873cd"}
Oct 29 22:29:02.779878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:02.779432    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8" event={"ID":"729aaaa7-6766-4d22-84c0-88bc54b2b74a","Type":"ContainerStarted","Data":"1eb4520b1a34a3260fab96797f40df7e63047d90afe4ffb05629cf8854c98d7b"}
Oct 29 22:29:02.781531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:02.781510    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6" event={"ID":"0debd5d9-d5f0-4c81-b35f-696613811010","Type":"ContainerStarted","Data":"14b04123a012203cdee7e2d85b927253def9a49d86626dd2a54a85a7a1649249"}
Oct 29 22:29:02.781652 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:02.781550    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6" event={"ID":"0debd5d9-d5f0-4c81-b35f-696613811010","Type":"ContainerStarted","Data":"0e0753029c50ec354e7e97039a6888debad3a3c1ded76c0859ec261695f049bb"}
Oct 29 22:29:02.820872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:02.820834    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8" podStartSLOduration=1.8208219780000001 podStartE2EDuration="1.820821978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:02.819954417 +0000 UTC m=+884.844730547" watchObservedRunningTime="2026-10-29 22:29:02.820821978 +0000 UTC m=+884.845598106"
Oct 29 22:29:03.556665 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:03.556621    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:03.789751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:03.789721    4860 generic.go:356] "Generic (PLEG): container finished" podID="729aaaa7-6766-4d22-84c0-88bc54b2b74a" containerID="4ae528fb19238c30386bac2d5d42808d26f87c0c97bf7faa9ab91932c42873cd" exitCode=0
Oct 29 22:29:03.789921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:03.789804    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8" event={"ID":"729aaaa7-6766-4d22-84c0-88bc54b2b74a","Type":"ContainerDied","Data":"4ae528fb19238c30386bac2d5d42808d26f87c0c97bf7faa9ab91932c42873cd"}
Oct 29 22:29:03.791538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:03.791511    4860 generic.go:356] "Generic (PLEG): container finished" podID="0debd5d9-d5f0-4c81-b35f-696613811010" containerID="14b04123a012203cdee7e2d85b927253def9a49d86626dd2a54a85a7a1649249" exitCode=0
Oct 29 22:29:03.791679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:03.791548    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6" event={"ID":"0debd5d9-d5f0-4c81-b35f-696613811010","Type":"ContainerDied","Data":"14b04123a012203cdee7e2d85b927253def9a49d86626dd2a54a85a7a1649249"}
Oct 29 22:29:03.948844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:03.948818    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:04.003144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.003116    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0debd5d9-d5f0-4c81-b35f-696613811010-kube-api-access-h6qx7\" (UniqueName: \"kubernetes.io/projected/0debd5d9-d5f0-4c81-b35f-696613811010-kube-api-access-h6qx7\") pod \"0debd5d9-d5f0-4c81-b35f-696613811010\" (UID: \"0debd5d9-d5f0-4c81-b35f-696613811010\") "
Oct 29 22:29:04.003304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.003215    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-openshift-ca\") pod \"0debd5d9-d5f0-4c81-b35f-696613811010\" (UID: \"0debd5d9-d5f0-4c81-b35f-696613811010\") "
Oct 29 22:29:04.003304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.003284    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-xks-ca\") pod \"0debd5d9-d5f0-4c81-b35f-696613811010\" (UID: \"0debd5d9-d5f0-4c81-b35f-696613811010\") "
Oct 29 22:29:04.003651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.003624    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-openshift-ca" pod "0debd5d9-d5f0-4c81-b35f-696613811010" (UID: "0debd5d9-d5f0-4c81-b35f-696613811010"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:29:04.003773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.003737    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:04.006002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.005972    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0debd5d9-d5f0-4c81-b35f-696613811010-kube-api-access-h6qx7" pod "0debd5d9-d5f0-4c81-b35f-696613811010" (UID: "0debd5d9-d5f0-4c81-b35f-696613811010"). InnerVolumeSpecName "kube-api-access-h6qx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:29:04.006135 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.005993    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-xks-ca" pod "0debd5d9-d5f0-4c81-b35f-696613811010" (UID: "0debd5d9-d5f0-4c81-b35f-696613811010"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:29:04.104549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.104492    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6qx7\" (UniqueName: \"kubernetes.io/projected/0debd5d9-d5f0-4c81-b35f-696613811010-kube-api-access-h6qx7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:04.104549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.104515    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0debd5d9-d5f0-4c81-b35f-696613811010-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:04.799548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.799518    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"
Oct 29 22:29:04.799954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.799558    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6" event={"ID":"0debd5d9-d5f0-4c81-b35f-696613811010","Type":"ContainerDied","Data":"0e0753029c50ec354e7e97039a6888debad3a3c1ded76c0859ec261695f049bb"}
Oct 29 22:29:04.799954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.799643    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e0753029c50ec354e7e97039a6888debad3a3c1ded76c0859ec261695f049bb"
Oct 29 22:29:04.946694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:04.946672    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:05.013051 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.013006    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-openshift-ca\") pod \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\" (UID: \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\") "
Oct 29 22:29:05.013209 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.013087    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-xks-ca\") pod \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\" (UID: \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\") "
Oct 29 22:29:05.013274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.013253    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729aaaa7-6766-4d22-84c0-88bc54b2b74a-kube-api-access-rmxgk\" (UniqueName: \"kubernetes.io/projected/729aaaa7-6766-4d22-84c0-88bc54b2b74a-kube-api-access-rmxgk\") pod \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\" (UID: \"729aaaa7-6766-4d22-84c0-88bc54b2b74a\") "
Oct 29 22:29:05.013472 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.013444    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-openshift-ca" pod "729aaaa7-6766-4d22-84c0-88bc54b2b74a" (UID: "729aaaa7-6766-4d22-84c0-88bc54b2b74a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:29:05.013841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.013810    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:05.015418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.015383    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-xks-ca" pod "729aaaa7-6766-4d22-84c0-88bc54b2b74a" (UID: "729aaaa7-6766-4d22-84c0-88bc54b2b74a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:29:05.015900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.015872    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729aaaa7-6766-4d22-84c0-88bc54b2b74a-kube-api-access-rmxgk" pod "729aaaa7-6766-4d22-84c0-88bc54b2b74a" (UID: "729aaaa7-6766-4d22-84c0-88bc54b2b74a"). InnerVolumeSpecName "kube-api-access-rmxgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:29:05.115093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.115037    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/729aaaa7-6766-4d22-84c0-88bc54b2b74a-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:05.115093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.115059    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmxgk\" (UniqueName: \"kubernetes.io/projected/729aaaa7-6766-4d22-84c0-88bc54b2b74a-kube-api-access-rmxgk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:05.346199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.346170    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:05.807899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.807869    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"
Oct 29 22:29:05.808274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.807905    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8" event={"ID":"729aaaa7-6766-4d22-84c0-88bc54b2b74a","Type":"ContainerDied","Data":"1eb4520b1a34a3260fab96797f40df7e63047d90afe4ffb05629cf8854c98d7b"}
Oct 29 22:29:05.808274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:05.807951    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eb4520b1a34a3260fab96797f40df7e63047d90afe4ffb05629cf8854c98d7b"
Oct 29 22:29:07.195364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:07.195139    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:08.956740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:08.956702    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:10.314629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:10.314596    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:10.603099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:10.602999    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:12.447916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.447882    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:12.524347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.524305    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"]
Oct 29 22:29:12.524902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.524885    4860 cpu_manager.go:414] "containerMap: removing container" podUID="729aaaa7-6766-4d22-84c0-88bc54b2b74a" containerName="revoke-keys"
Oct 29 22:29:12.524942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.524904    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729aaaa7-6766-4d22-84c0-88bc54b2b74a" containerName="revoke-keys"
Oct 29 22:29:12.524942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.524924    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0debd5d9-d5f0-4c81-b35f-696613811010" containerName="revoke-keys"
Oct 29 22:29:12.524942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.524929    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0debd5d9-d5f0-4c81-b35f-696613811010" containerName="revoke-keys"
Oct 29 22:29:12.525030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.525017    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0debd5d9-d5f0-4c81-b35f-696613811010" containerName="revoke-keys"
Oct 29 22:29:12.525076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.525031    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="729aaaa7-6766-4d22-84c0-88bc54b2b74a" containerName="revoke-keys"
Oct 29 22:29:12.528561 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.528541    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:12.555858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.555826    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:29:12.557150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.556633    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:29:12.557363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.556679    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:29:12.559871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.559848    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:29:12.567061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.567039    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"]
Oct 29 22:29:12.627996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.627967    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-698dw\" (UniqueName: \"kubernetes.io/projected/171fd0ec-db45-49be-af25-3cd16e968bc2-kube-api-access-698dw\") pod \"maas-api-revoke-sub-0c39cbe7-95ctc\" (UID: \"171fd0ec-db45-49be-af25-3cd16e968bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:12.628130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.628111    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c39cbe7-95ctc\" (UID: \"171fd0ec-db45-49be-af25-3cd16e968bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:12.628197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.628182    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c39cbe7-95ctc\" (UID: \"171fd0ec-db45-49be-af25-3cd16e968bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:12.729244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.729185    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-698dw\" (UniqueName: \"kubernetes.io/projected/171fd0ec-db45-49be-af25-3cd16e968bc2-kube-api-access-698dw\") pod \"maas-api-revoke-sub-0c39cbe7-95ctc\" (UID: \"171fd0ec-db45-49be-af25-3cd16e968bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:12.729362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.729280    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c39cbe7-95ctc\" (UID: \"171fd0ec-db45-49be-af25-3cd16e968bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:12.729362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.729304    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c39cbe7-95ctc\" (UID: \"171fd0ec-db45-49be-af25-3cd16e968bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:12.730005 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.729985    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c39cbe7-95ctc\" (UID: \"171fd0ec-db45-49be-af25-3cd16e968bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:12.731831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.731810    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c39cbe7-95ctc\" (UID: \"171fd0ec-db45-49be-af25-3cd16e968bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:12.751485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.751462    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-698dw\" (UniqueName: \"kubernetes.io/projected/171fd0ec-db45-49be-af25-3cd16e968bc2-kube-api-access-698dw\") pod \"maas-api-revoke-sub-0c39cbe7-95ctc\" (UID: \"171fd0ec-db45-49be-af25-3cd16e968bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:12.861691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:12.861670    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:13.157209 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:29:13.157162    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171fd0ec_db45_49be_af25_3cd16e968bc2.slice/crio-107cb816c5f437f52d2e677d2d2958a75c54c253c8dfc1ad58e4a7a4f0ecc0ce WatchSource:0}: Error finding container 107cb816c5f437f52d2e677d2d2958a75c54c253c8dfc1ad58e4a7a4f0ecc0ce: Status 404 returned error can't find the container with id 107cb816c5f437f52d2e677d2d2958a75c54c253c8dfc1ad58e4a7a4f0ecc0ce
Oct 29 22:29:13.159834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:13.159813    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"]
Oct 29 22:29:13.891714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:13.891639    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc" event={"ID":"171fd0ec-db45-49be-af25-3cd16e968bc2","Type":"ContainerStarted","Data":"4eb8898775461b38092c409da67b67b7533ea44018e31fb3dee6915f713f6804"}
Oct 29 22:29:13.891714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:13.891697    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc" event={"ID":"171fd0ec-db45-49be-af25-3cd16e968bc2","Type":"ContainerStarted","Data":"107cb816c5f437f52d2e677d2d2958a75c54c253c8dfc1ad58e4a7a4f0ecc0ce"}
Oct 29 22:29:13.935430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:13.935384    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc" podStartSLOduration=1.935371532 podStartE2EDuration="1.935371532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:13.934084675 +0000 UTC m=+895.958860827" watchObservedRunningTime="2026-10-29 22:29:13.935371532 +0000 UTC m=+895.960147662"
Oct 29 22:29:14.150916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.150836    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:14.715879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.715848    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"]
Oct 29 22:29:14.719844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.719827    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:14.740461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.740433    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"]
Oct 29 22:29:14.852629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.852601    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrw4z\" (UniqueName: \"kubernetes.io/projected/81017e47-c571-474e-a7ed-9aa0ca92a910-kube-api-access-rrw4z\") pod \"maas-api-revoke-sub-441251fc-p7sb4\" (UID: \"81017e47-c571-474e-a7ed-9aa0ca92a910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:14.852772 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.852754    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-441251fc-p7sb4\" (UID: \"81017e47-c571-474e-a7ed-9aa0ca92a910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:14.852840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.852790    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-441251fc-p7sb4\" (UID: \"81017e47-c571-474e-a7ed-9aa0ca92a910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:14.900487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.900461    4860 generic.go:356] "Generic (PLEG): container finished" podID="171fd0ec-db45-49be-af25-3cd16e968bc2" containerID="4eb8898775461b38092c409da67b67b7533ea44018e31fb3dee6915f713f6804" exitCode=0
Oct 29 22:29:14.900824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.900547    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc" event={"ID":"171fd0ec-db45-49be-af25-3cd16e968bc2","Type":"ContainerDied","Data":"4eb8898775461b38092c409da67b67b7533ea44018e31fb3dee6915f713f6804"}
Oct 29 22:29:14.954194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.954172    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-441251fc-p7sb4\" (UID: \"81017e47-c571-474e-a7ed-9aa0ca92a910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:14.954309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.954210    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-441251fc-p7sb4\" (UID: \"81017e47-c571-474e-a7ed-9aa0ca92a910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:14.954423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.954318    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrw4z\" (UniqueName: \"kubernetes.io/projected/81017e47-c571-474e-a7ed-9aa0ca92a910-kube-api-access-rrw4z\") pod \"maas-api-revoke-sub-441251fc-p7sb4\" (UID: \"81017e47-c571-474e-a7ed-9aa0ca92a910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:14.955071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.955042    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-441251fc-p7sb4\" (UID: \"81017e47-c571-474e-a7ed-9aa0ca92a910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:14.956771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.956748    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-441251fc-p7sb4\" (UID: \"81017e47-c571-474e-a7ed-9aa0ca92a910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:14.976294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:14.976240    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrw4z\" (UniqueName: \"kubernetes.io/projected/81017e47-c571-474e-a7ed-9aa0ca92a910-kube-api-access-rrw4z\") pod \"maas-api-revoke-sub-441251fc-p7sb4\" (UID: \"81017e47-c571-474e-a7ed-9aa0ca92a910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:15.033084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:15.033062    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:15.336636 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:29:15.336598    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81017e47_c571_474e_a7ed_9aa0ca92a910.slice/crio-6925bcd35839685a32c5faa1ab4be1edd53411f98e4156298789394045cda0a8 WatchSource:0}: Error finding container 6925bcd35839685a32c5faa1ab4be1edd53411f98e4156298789394045cda0a8: Status 404 returned error can't find the container with id 6925bcd35839685a32c5faa1ab4be1edd53411f98e4156298789394045cda0a8
Oct 29 22:29:15.340514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:15.340489    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"]
Oct 29 22:29:15.908468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:15.908433    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4" event={"ID":"81017e47-c571-474e-a7ed-9aa0ca92a910","Type":"ContainerStarted","Data":"812f1a0c4fc3c9fcb77e1139456d0e03423c54aae217b3d838753a8667386498"}
Oct 29 22:29:15.908848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:15.908492    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4" event={"ID":"81017e47-c571-474e-a7ed-9aa0ca92a910","Type":"ContainerStarted","Data":"6925bcd35839685a32c5faa1ab4be1edd53411f98e4156298789394045cda0a8"}
Oct 29 22:29:15.953913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:15.953861    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4" podStartSLOduration=1.9538468660000001 podStartE2EDuration="1.953846866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:15.952433052 +0000 UTC m=+897.977209215" watchObservedRunningTime="2026-10-29 22:29:15.953846866 +0000 UTC m=+897.978622996"
Oct 29 22:29:15.954559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:15.954518    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:16.062963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.062933    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:16.167586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.167554    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-xks-ca\") pod \"171fd0ec-db45-49be-af25-3cd16e968bc2\" (UID: \"171fd0ec-db45-49be-af25-3cd16e968bc2\") "
Oct 29 22:29:16.167754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.167716    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171fd0ec-db45-49be-af25-3cd16e968bc2-kube-api-access-698dw\" (UniqueName: \"kubernetes.io/projected/171fd0ec-db45-49be-af25-3cd16e968bc2-kube-api-access-698dw\") pod \"171fd0ec-db45-49be-af25-3cd16e968bc2\" (UID: \"171fd0ec-db45-49be-af25-3cd16e968bc2\") "
Oct 29 22:29:16.167851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.167836    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-openshift-ca\") pod \"171fd0ec-db45-49be-af25-3cd16e968bc2\" (UID: \"171fd0ec-db45-49be-af25-3cd16e968bc2\") "
Oct 29 22:29:16.168173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.168147    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-openshift-ca" pod "171fd0ec-db45-49be-af25-3cd16e968bc2" (UID: "171fd0ec-db45-49be-af25-3cd16e968bc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:29:16.168299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.168281    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:16.169859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.169832    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-xks-ca" pod "171fd0ec-db45-49be-af25-3cd16e968bc2" (UID: "171fd0ec-db45-49be-af25-3cd16e968bc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:29:16.170037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.170014    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171fd0ec-db45-49be-af25-3cd16e968bc2-kube-api-access-698dw" pod "171fd0ec-db45-49be-af25-3cd16e968bc2" (UID: "171fd0ec-db45-49be-af25-3cd16e968bc2"). InnerVolumeSpecName "kube-api-access-698dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:29:16.269475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.269449    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171fd0ec-db45-49be-af25-3cd16e968bc2-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:16.269551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.269480    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-698dw\" (UniqueName: \"kubernetes.io/projected/171fd0ec-db45-49be-af25-3cd16e968bc2-kube-api-access-698dw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:16.666566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.666487    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:16.916159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.916122    4860 generic.go:356] "Generic (PLEG): container finished" podID="81017e47-c571-474e-a7ed-9aa0ca92a910" containerID="812f1a0c4fc3c9fcb77e1139456d0e03423c54aae217b3d838753a8667386498" exitCode=0
Oct 29 22:29:16.916709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.916207    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4" event={"ID":"81017e47-c571-474e-a7ed-9aa0ca92a910","Type":"ContainerDied","Data":"812f1a0c4fc3c9fcb77e1139456d0e03423c54aae217b3d838753a8667386498"}
Oct 29 22:29:16.917756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.917740    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"
Oct 29 22:29:16.917854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.917736    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc" event={"ID":"171fd0ec-db45-49be-af25-3cd16e968bc2","Type":"ContainerDied","Data":"107cb816c5f437f52d2e677d2d2958a75c54c253c8dfc1ad58e4a7a4f0ecc0ce"}
Oct 29 22:29:16.917898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:16.917852    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="107cb816c5f437f52d2e677d2d2958a75c54c253c8dfc1ad58e4a7a4f0ecc0ce"
Oct 29 22:29:17.598623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:17.598589    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:18.057998 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:29:18.057962    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:29:18.067521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.067499    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:18.190501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.190469    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-xks-ca\") pod \"81017e47-c571-474e-a7ed-9aa0ca92a910\" (UID: \"81017e47-c571-474e-a7ed-9aa0ca92a910\") "
Oct 29 22:29:18.190675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.190524    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-openshift-ca\") pod \"81017e47-c571-474e-a7ed-9aa0ca92a910\" (UID: \"81017e47-c571-474e-a7ed-9aa0ca92a910\") "
Oct 29 22:29:18.190758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.190702    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81017e47-c571-474e-a7ed-9aa0ca92a910-kube-api-access-rrw4z\" (UniqueName: \"kubernetes.io/projected/81017e47-c571-474e-a7ed-9aa0ca92a910-kube-api-access-rrw4z\") pod \"81017e47-c571-474e-a7ed-9aa0ca92a910\" (UID: \"81017e47-c571-474e-a7ed-9aa0ca92a910\") "
Oct 29 22:29:18.191173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.191146    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-openshift-ca" pod "81017e47-c571-474e-a7ed-9aa0ca92a910" (UID: "81017e47-c571-474e-a7ed-9aa0ca92a910"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:29:18.193972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.193045    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-xks-ca" pod "81017e47-c571-474e-a7ed-9aa0ca92a910" (UID: "81017e47-c571-474e-a7ed-9aa0ca92a910"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:29:18.194923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.194882    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81017e47-c571-474e-a7ed-9aa0ca92a910-kube-api-access-rrw4z" pod "81017e47-c571-474e-a7ed-9aa0ca92a910" (UID: "81017e47-c571-474e-a7ed-9aa0ca92a910"). InnerVolumeSpecName "kube-api-access-rrw4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:29:18.228911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.228891    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:29:18.231021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.231003    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:29:18.291927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.291908    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:18.291927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.291929    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81017e47-c571-474e-a7ed-9aa0ca92a910-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:18.292069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.291938    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrw4z\" (UniqueName: \"kubernetes.io/projected/81017e47-c571-474e-a7ed-9aa0ca92a910-kube-api-access-rrw4z\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:18.931739 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.931711    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"
Oct 29 22:29:18.931739 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.931716    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4" event={"ID":"81017e47-c571-474e-a7ed-9aa0ca92a910","Type":"ContainerDied","Data":"6925bcd35839685a32c5faa1ab4be1edd53411f98e4156298789394045cda0a8"}
Oct 29 22:29:18.931923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:18.931764    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6925bcd35839685a32c5faa1ab4be1edd53411f98e4156298789394045cda0a8"
Oct 29 22:29:19.468216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:19.468172    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:21.151951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:21.151911    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:22.952814 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:22.952785    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:24.562510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:24.562480    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:25.028201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.028169    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"]
Oct 29 22:29:25.028920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.028899    4860 cpu_manager.go:414] "containerMap: removing container" podUID="81017e47-c571-474e-a7ed-9aa0ca92a910" containerName="revoke-keys"
Oct 29 22:29:25.028920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.028918    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81017e47-c571-474e-a7ed-9aa0ca92a910" containerName="revoke-keys"
Oct 29 22:29:25.029082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.028942    4860 cpu_manager.go:414] "containerMap: removing container" podUID="171fd0ec-db45-49be-af25-3cd16e968bc2" containerName="revoke-keys"
Oct 29 22:29:25.029082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.028949    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171fd0ec-db45-49be-af25-3cd16e968bc2" containerName="revoke-keys"
Oct 29 22:29:25.029141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.029082    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="81017e47-c571-474e-a7ed-9aa0ca92a910" containerName="revoke-keys"
Oct 29 22:29:25.029141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.029095    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="171fd0ec-db45-49be-af25-3cd16e968bc2" containerName="revoke-keys"
Oct 29 22:29:25.034056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.034039    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:25.050029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.050011    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"]
Oct 29 22:29:25.060354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.060307    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:29:25.060354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.060309    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:29:25.060489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.060342    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:29:25.065407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.065389    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:29:25.181100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.181074    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxv5p\" (UniqueName: \"kubernetes.io/projected/d00de079-cef7-43cb-ad71-c8109076fd07-kube-api-access-jxv5p\") pod \"maas-api-revoke-sub-d216c885-k4twd\" (UID: \"d00de079-cef7-43cb-ad71-c8109076fd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:25.181225 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.181111    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d216c885-k4twd\" (UID: \"d00de079-cef7-43cb-ad71-c8109076fd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:25.181276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.181257    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d216c885-k4twd\" (UID: \"d00de079-cef7-43cb-ad71-c8109076fd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:25.282051 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.281988    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxv5p\" (UniqueName: \"kubernetes.io/projected/d00de079-cef7-43cb-ad71-c8109076fd07-kube-api-access-jxv5p\") pod \"maas-api-revoke-sub-d216c885-k4twd\" (UID: \"d00de079-cef7-43cb-ad71-c8109076fd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:25.282051 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.282018    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d216c885-k4twd\" (UID: \"d00de079-cef7-43cb-ad71-c8109076fd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:25.282224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.282083    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d216c885-k4twd\" (UID: \"d00de079-cef7-43cb-ad71-c8109076fd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:25.282740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.282718    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d216c885-k4twd\" (UID: \"d00de079-cef7-43cb-ad71-c8109076fd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:25.284491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.284471    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d216c885-k4twd\" (UID: \"d00de079-cef7-43cb-ad71-c8109076fd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:25.303445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.303427    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxv5p\" (UniqueName: \"kubernetes.io/projected/d00de079-cef7-43cb-ad71-c8109076fd07-kube-api-access-jxv5p\") pod \"maas-api-revoke-sub-d216c885-k4twd\" (UID: \"d00de079-cef7-43cb-ad71-c8109076fd07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:25.347553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.347533    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:25.647785 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:29:25.647759    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00de079_cef7_43cb_ad71_c8109076fd07.slice/crio-68c7f01d7d9c84ae611a2ffe938a5b85d8c4c6fc9d63d104fe86393746ec37b9 WatchSource:0}: Error finding container 68c7f01d7d9c84ae611a2ffe938a5b85d8c4c6fc9d63d104fe86393746ec37b9: Status 404 returned error can't find the container with id 68c7f01d7d9c84ae611a2ffe938a5b85d8c4c6fc9d63d104fe86393746ec37b9
Oct 29 22:29:25.650763 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.650698    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"]
Oct 29 22:29:25.979920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:25.979891    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd" event={"ID":"d00de079-cef7-43cb-ad71-c8109076fd07","Type":"ContainerStarted","Data":"68c7f01d7d9c84ae611a2ffe938a5b85d8c4c6fc9d63d104fe86393746ec37b9"}
Oct 29 22:29:26.247802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:26.247722    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:26.988855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:26.988821    4860 generic.go:356] "Generic (PLEG): container finished" podID="d00de079-cef7-43cb-ad71-c8109076fd07" containerID="466869ed1c5639b12716135145eda11d00b0d3eedbafc9c773f881897aef136e" exitCode=0
Oct 29 22:29:26.989220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:26.988905    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd" event={"ID":"d00de079-cef7-43cb-ad71-c8109076fd07","Type":"ContainerDied","Data":"466869ed1c5639b12716135145eda11d00b0d3eedbafc9c773f881897aef136e"}
Oct 29 22:29:27.922127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:27.922088    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:28.176476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:28.176413    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:28.215344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:28.215248    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-openshift-ca\") pod \"d00de079-cef7-43cb-ad71-c8109076fd07\" (UID: \"d00de079-cef7-43cb-ad71-c8109076fd07\") "
Oct 29 22:29:28.215530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:28.215348    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-xks-ca\") pod \"d00de079-cef7-43cb-ad71-c8109076fd07\" (UID: \"d00de079-cef7-43cb-ad71-c8109076fd07\") "
Oct 29 22:29:28.215530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:28.215474    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00de079-cef7-43cb-ad71-c8109076fd07-kube-api-access-jxv5p\" (UniqueName: \"kubernetes.io/projected/d00de079-cef7-43cb-ad71-c8109076fd07-kube-api-access-jxv5p\") pod \"d00de079-cef7-43cb-ad71-c8109076fd07\" (UID: \"d00de079-cef7-43cb-ad71-c8109076fd07\") "
Oct 29 22:29:28.220345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:28.218069    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-openshift-ca" pod "d00de079-cef7-43cb-ad71-c8109076fd07" (UID: "d00de079-cef7-43cb-ad71-c8109076fd07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:29:28.290232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:28.289454    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-xks-ca" pod "d00de079-cef7-43cb-ad71-c8109076fd07" (UID: "d00de079-cef7-43cb-ad71-c8109076fd07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:29:28.290232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:28.289903    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00de079-cef7-43cb-ad71-c8109076fd07-kube-api-access-jxv5p" pod "d00de079-cef7-43cb-ad71-c8109076fd07" (UID: "d00de079-cef7-43cb-ad71-c8109076fd07"). InnerVolumeSpecName "kube-api-access-jxv5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:29:28.324805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:28.324772    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:28.325005 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:28.324988    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxv5p\" (UniqueName: \"kubernetes.io/projected/d00de079-cef7-43cb-ad71-c8109076fd07-kube-api-access-jxv5p\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:28.325121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:28.325108    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d00de079-cef7-43cb-ad71-c8109076fd07-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:29:29.006977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:29.006923    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd" event={"ID":"d00de079-cef7-43cb-ad71-c8109076fd07","Type":"ContainerDied","Data":"68c7f01d7d9c84ae611a2ffe938a5b85d8c4c6fc9d63d104fe86393746ec37b9"}
Oct 29 22:29:29.007166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:29.007034    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68c7f01d7d9c84ae611a2ffe938a5b85d8c4c6fc9d63d104fe86393746ec37b9"
Oct 29 22:29:29.007166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:29.007116    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"
Oct 29 22:29:29.678821 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:29.678788    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:31.350027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:31.349994    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:32.958449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:32.958414    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:34.764070 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:34.764031    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:34.903290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:34.903259    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:35.596731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.596700    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"]
Oct 29 22:29:35.597258 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.597244    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d00de079-cef7-43cb-ad71-c8109076fd07" containerName="revoke-keys"
Oct 29 22:29:35.597309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.597261    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00de079-cef7-43cb-ad71-c8109076fd07" containerName="revoke-keys"
Oct 29 22:29:35.597378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.597362    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00de079-cef7-43cb-ad71-c8109076fd07" containerName="revoke-keys"
Oct 29 22:29:35.600846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.600824    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.635500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.635475    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"]
Oct 29 22:29:35.704361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.704334    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.704508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.704377    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.704508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.704480    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mch5r\" (UniqueName: \"kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-kube-api-access-mch5r\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.704628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.704519    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb10043-0048-4595-bc98-b3333f888d5f-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.704628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.704541    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.704848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.704686    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.704848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.704758    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.704848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.704827    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb10043-0048-4595-bc98-b3333f888d5f-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.704968 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.704850    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.805962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.805936    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.805976    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.806017    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb10043-0048-4595-bc98-b3333f888d5f-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.806036    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.806059    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.806084    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.806143    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mch5r\" (UniqueName: \"kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-kube-api-access-mch5r\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.806172    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb10043-0048-4595-bc98-b3333f888d5f-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.806198    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.806516    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.806546    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.806650    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.806691    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.806952 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.806922    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb10043-0048-4595-bc98-b3333f888d5f-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.808610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.808590    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.808863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.808844    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb10043-0048-4595-bc98-b3333f888d5f-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.826419 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.826391    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"]
Oct 29 22:29:35.826884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.826862    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mch5r\" (UniqueName: \"kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-kube-api-access-mch5r\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.826989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.826879    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:35.831529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.831512    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:35.851534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.851511    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"]
Oct 29 22:29:35.907633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.907604    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:35.907746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.907651    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:35.907746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.907716    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g46rv\" (UniqueName: \"kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-kube-api-access-g46rv\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:35.907860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.907759    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:35.907921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.907893    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:35.907986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.907973    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:35.908053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.908035    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:35.908142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.908127    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:35.908201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.908159    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:35.915423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:35.915403    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:36.010369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.009775    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.010369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.009878    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.010369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.009942    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.010636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.010404    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.010636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.010518    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.010636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.010548    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.010636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.010589    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.010636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.010614    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.010636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.010629    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46rv\" (UniqueName: \"kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-kube-api-access-g46rv\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.010958 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.010664    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.011019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.010962    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.011551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.011528    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.012184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.012137    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.012495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.012471    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.013427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.013399    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.014391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.014306    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.031494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.031466    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g46rv\" (UniqueName: \"kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-kube-api-access-g46rv\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.031719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.031704    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.155387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.155297    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:36.232180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.232153    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"]
Oct 29 22:29:36.234363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.234308    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:29:36.234450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.234409    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:29:36.234503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.234449    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:29:36.459585 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:29:36.459559    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8f3210_ba1b_4981_bd2a_8ea5475773d5.slice/crio-29f5311a91b61c2ee2de8698f0780dd14b3bbbcfef5708be4adefcd1108f232a WatchSource:0}: Error finding container 29f5311a91b61c2ee2de8698f0780dd14b3bbbcfef5708be4adefcd1108f232a: Status 404 returned error can't find the container with id 29f5311a91b61c2ee2de8698f0780dd14b3bbbcfef5708be4adefcd1108f232a
Oct 29 22:29:36.462921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.462878    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:29:36.463037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.462976    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:29:36.463037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.463024    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:29:36.463445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.463421    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"]
Oct 29 22:29:36.733528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.733433    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:36.861400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:36.861362    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:37.070397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:37.069187    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" event={"ID":"ee8f3210-ba1b-4981-bd2a-8ea5475773d5","Type":"ContainerStarted","Data":"0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65"}
Oct 29 22:29:37.070397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:37.069257    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" event={"ID":"ee8f3210-ba1b-4981-bd2a-8ea5475773d5","Type":"ContainerStarted","Data":"29f5311a91b61c2ee2de8698f0780dd14b3bbbcfef5708be4adefcd1108f232a"}
Oct 29 22:29:37.109366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:37.089967    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s" event={"ID":"9bb10043-0048-4595-bc98-b3333f888d5f","Type":"ContainerStarted","Data":"c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535"}
Oct 29 22:29:37.109366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:37.090031    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s" event={"ID":"9bb10043-0048-4595-bc98-b3333f888d5f","Type":"ContainerStarted","Data":"89d723ad2769e8fa2acef75a88fc794d01619975b1071c1dc55994ac2bf48c8a"}
Oct 29 22:29:37.140997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:37.140929    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" podStartSLOduration=2.140909523 podStartE2EDuration="2.140909523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:37.125249719 +0000 UTC m=+919.150025875" watchObservedRunningTime="2026-10-29 22:29:37.140909523 +0000 UTC m=+919.165685655"
Oct 29 22:29:37.155488 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:37.155446    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:37.157726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:37.157694    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:29:37.157840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:37.157762    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 22:29:37.223350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:37.222114    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s" podStartSLOduration=2.222095081 podStartE2EDuration="2.222095081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:37.194881478 +0000 UTC m=+919.219657626" watchObservedRunningTime="2026-10-29 22:29:37.222095081 +0000 UTC m=+919.246871212"
Oct 29 22:29:37.921347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:37.920693    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:37.932351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:37.927210    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:29:37.932351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:37.927268    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s" podUID="9bb10043-0048-4595-bc98-b3333f888d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 29 22:29:38.156399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:38.156362    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:29:38.156612 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:38.156420    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 22:29:38.921503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:38.921453    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:29:38.922004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:38.921521    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s" podUID="9bb10043-0048-4595-bc98-b3333f888d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 29 22:29:39.156070 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:39.156038    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:29:39.156213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:39.156103    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 22:29:39.388791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:39.388756    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:39.917454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:39.917412    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:29:39.917625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:39.917493    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s" podUID="9bb10043-0048-4595-bc98-b3333f888d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 29 22:29:40.155626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:40.155587    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:29:40.156024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:40.155657    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 22:29:40.920350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:40.919755    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:29:40.920350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:40.919833    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s" podUID="9bb10043-0048-4595-bc98-b3333f888d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 29 22:29:41.156193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:41.156148    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:29:41.156193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:41.156194    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 22:29:41.719265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:41.719193    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:41.917796 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:41.917729    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:29:41.917972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:41.917875    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s" podUID="9bb10043-0048-4595-bc98-b3333f888d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 29 22:29:42.155751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:42.155664    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:29:42.155751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:42.155726    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 22:29:43.013835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:43.012882    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:43.013835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:43.013779    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:43.017061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:43.017041    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:29:43.161350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:43.160826    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:29:43.161350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:43.160878    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 22:29:44.160517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:44.160488    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:44.161041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:44.161025    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:44.161682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:44.161664    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:29:44.201731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:44.201709    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:46.820316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:46.820284    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:48.055556 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:29:48.055516    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:29:49.028274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:49.028229    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:51.043553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:51.043519    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:51.679361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:51.679305    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:52.805354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:52.805289    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:54.638405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:54.638372    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:56.325921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:56.325884    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:57.885686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:57.885650    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:29:59.507436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:29:59.507405    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:00.157033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.156993    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"]
Oct 29 22:30:00.164014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.163986    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"]
Oct 29 22:30:00.164154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.164116    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"
Oct 29 22:30:00.195646 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.195612    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:30:00.198998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.195994    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:30:00.221385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.221029    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"]
Oct 29 22:30:00.228013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.227989    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"]
Oct 29 22:30:00.231642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.231619    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"
Oct 29 22:30:00.232485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.232463    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"
Oct 29 22:30:00.233026 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.232999    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"]
Oct 29 22:30:00.238393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.238138    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"
Oct 29 22:30:00.258356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.258336    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"]
Oct 29 22:30:00.258789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.258772    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"]
Oct 29 22:30:00.259608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.259594    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"]
Oct 29 22:30:00.259726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.259718    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"]
Oct 29 22:30:00.259810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.259804    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"]
Oct 29 22:30:00.260186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.258621    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"
Oct 29 22:30:00.279260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.279241    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7843bdf-8d86-4901-988c-af9753f6df6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw\" (UID: \"f7843bdf-8d86-4901-988c-af9753f6df6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"
Oct 29 22:30:00.279367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.279273    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6qdf\" (UniqueName: \"kubernetes.io/projected/f7843bdf-8d86-4901-988c-af9753f6df6b-kube-api-access-l6qdf\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw\" (UID: \"f7843bdf-8d86-4901-988c-af9753f6df6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"
Oct 29 22:30:00.280143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.280125    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:30:00.380874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.380842    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647ac886-ea39-4651-9903-a242645e1dcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9\" (UID: \"647ac886-ea39-4651-9903-a242645e1dcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"
Oct 29 22:30:00.381064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.380888    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcstq\" (UniqueName: \"kubernetes.io/projected/b8a7d489-a427-427d-999e-0e0268c79457-kube-api-access-mcstq\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b\" (UID: \"b8a7d489-a427-427d-999e-0e0268c79457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"
Oct 29 22:30:00.381064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.380918    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f90084-0afe-430f-a9a9-f7f0775e02f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54\" (UID: \"e5f90084-0afe-430f-a9a9-f7f0775e02f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"
Oct 29 22:30:00.381221 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.381194    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0744a866-252d-4670-97aa-b11148022f06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr\" (UID: \"0744a866-252d-4670-97aa-b11148022f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"
Oct 29 22:30:00.381286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.381261    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jbjc\" (UniqueName: \"kubernetes.io/projected/0744a866-252d-4670-97aa-b11148022f06-kube-api-access-5jbjc\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr\" (UID: \"0744a866-252d-4670-97aa-b11148022f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"
Oct 29 22:30:00.381386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.381368    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7843bdf-8d86-4901-988c-af9753f6df6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw\" (UID: \"f7843bdf-8d86-4901-988c-af9753f6df6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"
Oct 29 22:30:00.381442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.381414    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qdf\" (UniqueName: \"kubernetes.io/projected/f7843bdf-8d86-4901-988c-af9753f6df6b-kube-api-access-l6qdf\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw\" (UID: \"f7843bdf-8d86-4901-988c-af9753f6df6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"
Oct 29 22:30:00.381479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.381452    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8a7d489-a427-427d-999e-0e0268c79457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b\" (UID: \"b8a7d489-a427-427d-999e-0e0268c79457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"
Oct 29 22:30:00.381537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.381496    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnz4\" (UniqueName: \"kubernetes.io/projected/647ac886-ea39-4651-9903-a242645e1dcd-kube-api-access-9qnz4\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9\" (UID: \"647ac886-ea39-4651-9903-a242645e1dcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"
Oct 29 22:30:00.381591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.381539    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c554w\" (UniqueName: \"kubernetes.io/projected/e5f90084-0afe-430f-a9a9-f7f0775e02f2-kube-api-access-c554w\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54\" (UID: \"e5f90084-0afe-430f-a9a9-f7f0775e02f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"
Oct 29 22:30:00.382212 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.382187    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7843bdf-8d86-4901-988c-af9753f6df6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw\" (UID: \"f7843bdf-8d86-4901-988c-af9753f6df6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"
Oct 29 22:30:00.401984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.401949    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qdf\" (UniqueName: \"kubernetes.io/projected/f7843bdf-8d86-4901-988c-af9753f6df6b-kube-api-access-l6qdf\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw\" (UID: \"f7843bdf-8d86-4901-988c-af9753f6df6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"
Oct 29 22:30:00.484525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.482492    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8a7d489-a427-427d-999e-0e0268c79457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b\" (UID: \"b8a7d489-a427-427d-999e-0e0268c79457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"
Oct 29 22:30:00.484525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.482567    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnz4\" (UniqueName: \"kubernetes.io/projected/647ac886-ea39-4651-9903-a242645e1dcd-kube-api-access-9qnz4\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9\" (UID: \"647ac886-ea39-4651-9903-a242645e1dcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"
Oct 29 22:30:00.484525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.482613    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c554w\" (UniqueName: \"kubernetes.io/projected/e5f90084-0afe-430f-a9a9-f7f0775e02f2-kube-api-access-c554w\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54\" (UID: \"e5f90084-0afe-430f-a9a9-f7f0775e02f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"
Oct 29 22:30:00.484525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.482657    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647ac886-ea39-4651-9903-a242645e1dcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9\" (UID: \"647ac886-ea39-4651-9903-a242645e1dcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"
Oct 29 22:30:00.484525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.482692    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcstq\" (UniqueName: \"kubernetes.io/projected/b8a7d489-a427-427d-999e-0e0268c79457-kube-api-access-mcstq\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b\" (UID: \"b8a7d489-a427-427d-999e-0e0268c79457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"
Oct 29 22:30:00.484525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.482720    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f90084-0afe-430f-a9a9-f7f0775e02f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54\" (UID: \"e5f90084-0afe-430f-a9a9-f7f0775e02f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"
Oct 29 22:30:00.484525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.482867    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0744a866-252d-4670-97aa-b11148022f06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr\" (UID: \"0744a866-252d-4670-97aa-b11148022f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"
Oct 29 22:30:00.484525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.482920    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jbjc\" (UniqueName: \"kubernetes.io/projected/0744a866-252d-4670-97aa-b11148022f06-kube-api-access-5jbjc\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr\" (UID: \"0744a866-252d-4670-97aa-b11148022f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"
Oct 29 22:30:00.484525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.484203    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647ac886-ea39-4651-9903-a242645e1dcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9\" (UID: \"647ac886-ea39-4651-9903-a242645e1dcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"
Oct 29 22:30:00.485186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.485162    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f90084-0afe-430f-a9a9-f7f0775e02f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54\" (UID: \"e5f90084-0afe-430f-a9a9-f7f0775e02f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"
Oct 29 22:30:00.485946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.485919    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0744a866-252d-4670-97aa-b11148022f06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr\" (UID: \"0744a866-252d-4670-97aa-b11148022f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"
Oct 29 22:30:00.486978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.486954    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8a7d489-a427-427d-999e-0e0268c79457-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b\" (UID: \"b8a7d489-a427-427d-999e-0e0268c79457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"
Oct 29 22:30:00.518494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.514514    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"
Oct 29 22:30:00.535973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.532921    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c554w\" (UniqueName: \"kubernetes.io/projected/e5f90084-0afe-430f-a9a9-f7f0775e02f2-kube-api-access-c554w\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54\" (UID: \"e5f90084-0afe-430f-a9a9-f7f0775e02f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"
Oct 29 22:30:00.535973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.534380    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnz4\" (UniqueName: \"kubernetes.io/projected/647ac886-ea39-4651-9903-a242645e1dcd-kube-api-access-9qnz4\") pod \"maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9\" (UID: \"647ac886-ea39-4651-9903-a242645e1dcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"
Oct 29 22:30:00.535973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.535083    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcstq\" (UniqueName: \"kubernetes.io/projected/b8a7d489-a427-427d-999e-0e0268c79457-kube-api-access-mcstq\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b\" (UID: \"b8a7d489-a427-427d-999e-0e0268c79457\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"
Oct 29 22:30:00.536249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.536037    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jbjc\" (UniqueName: \"kubernetes.io/projected/0744a866-252d-4670-97aa-b11148022f06-kube-api-access-5jbjc\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr\" (UID: \"0744a866-252d-4670-97aa-b11148022f06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"
Oct 29 22:30:00.561362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.561300    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"
Oct 29 22:30:00.589593 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.589511    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"
Oct 29 22:30:00.589924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.589885    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"
Oct 29 22:30:00.596389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:00.592924    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"
Oct 29 22:30:01.141088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.141045    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"]
Oct 29 22:30:01.151001 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.150970    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"]
Oct 29 22:30:01.157081 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:30:01.157054    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a7d489_a427_427d_999e_0e0268c79457.slice/crio-b30c96b66cbf87c40f43f7d449a48d2639d94fe6b15402cb38920a4b9d028edc WatchSource:0}: Error finding container b30c96b66cbf87c40f43f7d449a48d2639d94fe6b15402cb38920a4b9d028edc: Status 404 returned error can't find the container with id b30c96b66cbf87c40f43f7d449a48d2639d94fe6b15402cb38920a4b9d028edc
Oct 29 22:30:01.213673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.213638    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"]
Oct 29 22:30:01.286399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.286367    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"]
Oct 29 22:30:01.317294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.317258    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"]
Oct 29 22:30:01.326504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.326421    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw" event={"ID":"f7843bdf-8d86-4901-988c-af9753f6df6b","Type":"ContainerStarted","Data":"3d5e3ec70a714a39cc3439f0b2a0b60d540a1f2a07a0ce33898cba8d5982e3e7"}
Oct 29 22:30:01.328152 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.328126    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b" event={"ID":"b8a7d489-a427-427d-999e-0e0268c79457","Type":"ContainerStarted","Data":"b30c96b66cbf87c40f43f7d449a48d2639d94fe6b15402cb38920a4b9d028edc"}
Oct 29 22:30:01.329649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.329628    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54" event={"ID":"e5f90084-0afe-430f-a9a9-f7f0775e02f2","Type":"ContainerStarted","Data":"3a0a6870c91659289fbd445f03a1417c83c007f4da54e764ecbe454a9a9ece25"}
Oct 29 22:30:01.433653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.433621    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:01.878136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.878104    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"]
Oct 29 22:30:01.884121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.884099    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:01.904739 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.904708    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"]
Oct 29 22:30:01.912658 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.912615    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-08ebee\"/\"distinct-w4-08ebee-kserve-self-signed-certs\""
Oct 29 22:30:01.955679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:01.955644    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:02.027174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.027134    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-home\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.027377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.027181    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6gqq\" (UniqueName: \"kubernetes.io/projected/c03c61aa-d816-451a-80f0-c636cf8800a6-kube-api-access-b6gqq\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.027377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.027259    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c03c61aa-d816-451a-80f0-c636cf8800a6-tls-certs\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.027377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.027364    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-tmp-dir\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.027511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.027481    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-dshm\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.027551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.027523    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-model-cache\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.128557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.128469    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-home\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.128557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.128525    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6gqq\" (UniqueName: \"kubernetes.io/projected/c03c61aa-d816-451a-80f0-c636cf8800a6-kube-api-access-b6gqq\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.128816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.128572    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c03c61aa-d816-451a-80f0-c636cf8800a6-tls-certs\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.128816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.128613    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-tmp-dir\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.128816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.128742    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-dshm\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.128816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.128795    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-model-cache\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.129027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.128953    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-home\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.129173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.129151    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-model-cache\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.129650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.129606    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-tmp-dir\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.142071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.142018    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-dshm\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.150290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.150254    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c03c61aa-d816-451a-80f0-c636cf8800a6-tls-certs\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.157306 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.157268    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6gqq\" (UniqueName: \"kubernetes.io/projected/c03c61aa-d816-451a-80f0-c636cf8800a6-kube-api-access-b6gqq\") pod \"distinct-w4-08ebee-kserve-65f799894d-jgvpg\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") " pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.203158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.203132    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:02.339514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.339484    4860 generic.go:356] "Generic (PLEG): container finished" podID="b8a7d489-a427-427d-999e-0e0268c79457" containerID="e7ccdb308d6e794a899b44c296821bc7522a642ee92a5ee8c16cf2c9000ac210" exitCode=60
Oct 29 22:30:02.339659 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.339565    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b" event={"ID":"b8a7d489-a427-427d-999e-0e0268c79457","Type":"ContainerDied","Data":"e7ccdb308d6e794a899b44c296821bc7522a642ee92a5ee8c16cf2c9000ac210"}
Oct 29 22:30:02.339841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.339826    4860 scope.go:122] "RemoveContainer" containerID="e7ccdb308d6e794a899b44c296821bc7522a642ee92a5ee8c16cf2c9000ac210"
Oct 29 22:30:02.341516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.341378    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54" event={"ID":"e5f90084-0afe-430f-a9a9-f7f0775e02f2","Type":"ContainerStarted","Data":"2b1627848b3975a74a83fc7974ea224ccdebf6a35b6cf26a42053f579404b368"}
Oct 29 22:30:02.345792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.345749    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr" event={"ID":"0744a866-252d-4670-97aa-b11148022f06","Type":"ContainerStarted","Data":"ffe7906f3604fa54f7fc1834d600b6575225307c1daf998557e20fc6e0bf60ba"}
Oct 29 22:30:02.345930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.345810    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr" event={"ID":"0744a866-252d-4670-97aa-b11148022f06","Type":"ContainerStarted","Data":"a9d995273cb0f02990c8a837a032202e16e118e71ba37da1c1fb6fd99d9a8c69"}
Oct 29 22:30:02.348093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.348073    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" event={"ID":"647ac886-ea39-4651-9903-a242645e1dcd","Type":"ContainerStarted","Data":"39c09bae2f5bb591109a38368e0c345d5c76f65bd9a795ee2fe97f38b542f54e"}
Oct 29 22:30:02.348166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.348109    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" event={"ID":"647ac886-ea39-4651-9903-a242645e1dcd","Type":"ContainerStarted","Data":"fac7d072ca0bd5f92ea2d2fe0ddb9952891e7ad6d8c2bb8ab9ab0f4e4ebbda87"}
Oct 29 22:30:02.349924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.349900    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw" event={"ID":"f7843bdf-8d86-4901-988c-af9753f6df6b","Type":"ContainerStarted","Data":"4564948b2e91c6a80e8ac9aca34414dad4b720d077c5978bb85f1fc43a03945c"}
Oct 29 22:30:02.421254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.421205    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw" podStartSLOduration=2.421191797 podStartE2EDuration="2.421191797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:02.42035753 +0000 UTC m=+944.445133660" watchObservedRunningTime="2026-10-29 22:30:02.421191797 +0000 UTC m=+944.445967927"
Oct 29 22:30:02.462317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.462267    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54" podStartSLOduration=2.462253696 podStartE2EDuration="2.462253696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:02.461129275 +0000 UTC m=+944.485905410" watchObservedRunningTime="2026-10-29 22:30:02.462253696 +0000 UTC m=+944.487029826"
Oct 29 22:30:02.503638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.503585    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr" podStartSLOduration=2.503570749 podStartE2EDuration="2.503570749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:02.500979906 +0000 UTC m=+944.525756036" watchObservedRunningTime="2026-10-29 22:30:02.503570749 +0000 UTC m=+944.528346929"
Oct 29 22:30:02.512737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.512710    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"]
Oct 29 22:30:02.534293 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:30:02.534258    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc03c61aa_d816_451a_80f0_c636cf8800a6.slice/crio-7b7bd084034297c601fab789db7109b9877667b7d3368ff0e84592916d38839c WatchSource:0}: Error finding container 7b7bd084034297c601fab789db7109b9877667b7d3368ff0e84592916d38839c: Status 404 returned error can't find the container with id 7b7bd084034297c601fab789db7109b9877667b7d3368ff0e84592916d38839c
Oct 29 22:30:02.543684 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:02.543642    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" podStartSLOduration=2.54362867 podStartE2EDuration="2.54362867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:02.540681439 +0000 UTC m=+944.565457569" watchObservedRunningTime="2026-10-29 22:30:02.54362867 +0000 UTC m=+944.568404800"
Oct 29 22:30:03.131043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.131012    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:03.358633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.358596    4860 generic.go:356] "Generic (PLEG): container finished" podID="f7843bdf-8d86-4901-988c-af9753f6df6b" containerID="4564948b2e91c6a80e8ac9aca34414dad4b720d077c5978bb85f1fc43a03945c" exitCode=60
Oct 29 22:30:03.358824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.358681    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw" event={"ID":"f7843bdf-8d86-4901-988c-af9753f6df6b","Type":"ContainerDied","Data":"4564948b2e91c6a80e8ac9aca34414dad4b720d077c5978bb85f1fc43a03945c"}
Oct 29 22:30:03.358957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.358929    4860 scope.go:122] "RemoveContainer" containerID="4564948b2e91c6a80e8ac9aca34414dad4b720d077c5978bb85f1fc43a03945c"
Oct 29 22:30:03.360528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.360509    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" event={"ID":"c03c61aa-d816-451a-80f0-c636cf8800a6","Type":"ContainerStarted","Data":"df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e"}
Oct 29 22:30:03.360588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.360543    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" event={"ID":"c03c61aa-d816-451a-80f0-c636cf8800a6","Type":"ContainerStarted","Data":"7b7bd084034297c601fab789db7109b9877667b7d3368ff0e84592916d38839c"}
Oct 29 22:30:03.362623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.362603    4860 generic.go:356] "Generic (PLEG): container finished" podID="b8a7d489-a427-427d-999e-0e0268c79457" containerID="55b1d44caab9ad33799b774713fab439f849ec6adad23b1b55b2b55bad55f54e" exitCode=60
Oct 29 22:30:03.362732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.362653    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b" event={"ID":"b8a7d489-a427-427d-999e-0e0268c79457","Type":"ContainerDied","Data":"55b1d44caab9ad33799b774713fab439f849ec6adad23b1b55b2b55bad55f54e"}
Oct 29 22:30:03.362732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.362707    4860 scope.go:122] "RemoveContainer" containerID="e7ccdb308d6e794a899b44c296821bc7522a642ee92a5ee8c16cf2c9000ac210"
Oct 29 22:30:03.362917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.362899    4860 scope.go:122] "RemoveContainer" containerID="55b1d44caab9ad33799b774713fab439f849ec6adad23b1b55b2b55bad55f54e"
Oct 29 22:30:03.363137 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:03.363115    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b_odh-ai-gateway-infra(b8a7d489-a427-427d-999e-0e0268c79457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b" podUID="b8a7d489-a427-427d-999e-0e0268c79457"
Oct 29 22:30:03.364219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.364198    4860 generic.go:356] "Generic (PLEG): container finished" podID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" containerID="2b1627848b3975a74a83fc7974ea224ccdebf6a35b6cf26a42053f579404b368" exitCode=60
Oct 29 22:30:03.364316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.364223    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54" event={"ID":"e5f90084-0afe-430f-a9a9-f7f0775e02f2","Type":"ContainerDied","Data":"2b1627848b3975a74a83fc7974ea224ccdebf6a35b6cf26a42053f579404b368"}
Oct 29 22:30:03.364487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.364471    4860 scope.go:122] "RemoveContainer" containerID="2b1627848b3975a74a83fc7974ea224ccdebf6a35b6cf26a42053f579404b368"
Oct 29 22:30:03.365994 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.365974    4860 generic.go:356] "Generic (PLEG): container finished" podID="0744a866-252d-4670-97aa-b11148022f06" containerID="ffe7906f3604fa54f7fc1834d600b6575225307c1daf998557e20fc6e0bf60ba" exitCode=60
Oct 29 22:30:03.366080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.366051    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr" event={"ID":"0744a866-252d-4670-97aa-b11148022f06","Type":"ContainerDied","Data":"ffe7906f3604fa54f7fc1834d600b6575225307c1daf998557e20fc6e0bf60ba"}
Oct 29 22:30:03.366220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.366207    4860 scope.go:122] "RemoveContainer" containerID="ffe7906f3604fa54f7fc1834d600b6575225307c1daf998557e20fc6e0bf60ba"
Oct 29 22:30:03.367718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.367696    4860 generic.go:356] "Generic (PLEG): container finished" podID="647ac886-ea39-4651-9903-a242645e1dcd" containerID="39c09bae2f5bb591109a38368e0c345d5c76f65bd9a795ee2fe97f38b542f54e" exitCode=60
Oct 29 22:30:03.367817 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.367716    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" event={"ID":"647ac886-ea39-4651-9903-a242645e1dcd","Type":"ContainerDied","Data":"39c09bae2f5bb591109a38368e0c345d5c76f65bd9a795ee2fe97f38b542f54e"}
Oct 29 22:30:03.368087 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.368068    4860 scope.go:122] "RemoveContainer" containerID="39c09bae2f5bb591109a38368e0c345d5c76f65bd9a795ee2fe97f38b542f54e"
Oct 29 22:30:03.607553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:03.607511    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podStartSLOduration=2.607497746 podStartE2EDuration="2.607497746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:03.605436232 +0000 UTC m=+945.630212401" watchObservedRunningTime="2026-10-29 22:30:03.607497746 +0000 UTC m=+945.632273931"
Oct 29 22:30:04.204237 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.204199    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:04.379851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.379813    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" event={"ID":"647ac886-ea39-4651-9903-a242645e1dcd","Type":"ContainerStarted","Data":"ec1fd491629bc8cbc210553d6109fcd8e9e19858ef8b72d759eaf203a7b415a9"}
Oct 29 22:30:04.380130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.380113    4860 scope.go:122] "RemoveContainer" containerID="ec1fd491629bc8cbc210553d6109fcd8e9e19858ef8b72d759eaf203a7b415a9"
Oct 29 22:30:04.380455 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:04.380431    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9_odh-ai-gateway-infra(647ac886-ea39-4651-9903-a242645e1dcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" podUID="647ac886-ea39-4651-9903-a242645e1dcd"
Oct 29 22:30:04.381920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.381895    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw" event={"ID":"f7843bdf-8d86-4901-988c-af9753f6df6b","Type":"ContainerStarted","Data":"a20cb67ac71f9d6c68e18f07875385b07a4612276eb00bef0db1baca44d0449f"}
Oct 29 22:30:04.383797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.383779    4860 scope.go:122] "RemoveContainer" containerID="55b1d44caab9ad33799b774713fab439f849ec6adad23b1b55b2b55bad55f54e"
Oct 29 22:30:04.384022 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:04.384004    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b_odh-ai-gateway-infra(b8a7d489-a427-427d-999e-0e0268c79457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b" podUID="b8a7d489-a427-427d-999e-0e0268c79457"
Oct 29 22:30:04.385279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.385260    4860 generic.go:356] "Generic (PLEG): container finished" podID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" containerID="c386ea7da6e5ebff0cc157b820ba2f7fb5f6de7ffe57565897dc1f3c350a111f" exitCode=60
Oct 29 22:30:04.385512 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.385398    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54" event={"ID":"e5f90084-0afe-430f-a9a9-f7f0775e02f2","Type":"ContainerDied","Data":"c386ea7da6e5ebff0cc157b820ba2f7fb5f6de7ffe57565897dc1f3c350a111f"}
Oct 29 22:30:04.385512 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.385443    4860 scope.go:122] "RemoveContainer" containerID="2b1627848b3975a74a83fc7974ea224ccdebf6a35b6cf26a42053f579404b368"
Oct 29 22:30:04.385648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.385534    4860 scope.go:122] "RemoveContainer" containerID="c386ea7da6e5ebff0cc157b820ba2f7fb5f6de7ffe57565897dc1f3c350a111f"
Oct 29 22:30:04.385755 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:04.385732    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54_odh-ai-gateway-infra(e5f90084-0afe-430f-a9a9-f7f0775e02f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2"
Oct 29 22:30:04.392587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.392562    4860 generic.go:356] "Generic (PLEG): container finished" podID="0744a866-252d-4670-97aa-b11148022f06" containerID="ce486ff2b9a60928bad296b95580299539724dca6d429de1bed778cc842a3845" exitCode=60
Oct 29 22:30:04.392694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.392637    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr" event={"ID":"0744a866-252d-4670-97aa-b11148022f06","Type":"ContainerDied","Data":"ce486ff2b9a60928bad296b95580299539724dca6d429de1bed778cc842a3845"}
Oct 29 22:30:04.392870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.392854    4860 scope.go:122] "RemoveContainer" containerID="ce486ff2b9a60928bad296b95580299539724dca6d429de1bed778cc842a3845"
Oct 29 22:30:04.393089 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:04.393071    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr_odh-ai-gateway-infra(0744a866-252d-4670-97aa-b11148022f06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr" podUID="0744a866-252d-4670-97aa-b11148022f06"
Oct 29 22:30:04.508199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.508172    4860 scope.go:122] "RemoveContainer" containerID="ffe7906f3604fa54f7fc1834d600b6575225307c1daf998557e20fc6e0bf60ba"
Oct 29 22:30:04.832751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:04.832679    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:05.402067 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:05.402037    4860 scope.go:122] "RemoveContainer" containerID="c386ea7da6e5ebff0cc157b820ba2f7fb5f6de7ffe57565897dc1f3c350a111f"
Oct 29 22:30:05.402558 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:05.402358    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54_odh-ai-gateway-infra(e5f90084-0afe-430f-a9a9-f7f0775e02f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2"
Oct 29 22:30:05.403731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:05.403702    4860 scope.go:122] "RemoveContainer" containerID="ce486ff2b9a60928bad296b95580299539724dca6d429de1bed778cc842a3845"
Oct 29 22:30:05.404035 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:05.404012    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr_odh-ai-gateway-infra(0744a866-252d-4670-97aa-b11148022f06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr" podUID="0744a866-252d-4670-97aa-b11148022f06"
Oct 29 22:30:05.405234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:05.405211    4860 generic.go:356] "Generic (PLEG): container finished" podID="647ac886-ea39-4651-9903-a242645e1dcd" containerID="ec1fd491629bc8cbc210553d6109fcd8e9e19858ef8b72d759eaf203a7b415a9" exitCode=60
Oct 29 22:30:05.405310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:05.405284    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" event={"ID":"647ac886-ea39-4651-9903-a242645e1dcd","Type":"ContainerDied","Data":"ec1fd491629bc8cbc210553d6109fcd8e9e19858ef8b72d759eaf203a7b415a9"}
Oct 29 22:30:05.405386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:05.405363    4860 scope.go:122] "RemoveContainer" containerID="39c09bae2f5bb591109a38368e0c345d5c76f65bd9a795ee2fe97f38b542f54e"
Oct 29 22:30:05.405520 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:05.405503    4860 scope.go:122] "RemoveContainer" containerID="ec1fd491629bc8cbc210553d6109fcd8e9e19858ef8b72d759eaf203a7b415a9"
Oct 29 22:30:05.405794 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:05.405773    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9_odh-ai-gateway-infra(647ac886-ea39-4651-9903-a242645e1dcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" podUID="647ac886-ea39-4651-9903-a242645e1dcd"
Oct 29 22:30:05.407309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:05.407289    4860 generic.go:356] "Generic (PLEG): container finished" podID="f7843bdf-8d86-4901-988c-af9753f6df6b" containerID="a20cb67ac71f9d6c68e18f07875385b07a4612276eb00bef0db1baca44d0449f" exitCode=60
Oct 29 22:30:05.407428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:05.407359    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw" event={"ID":"f7843bdf-8d86-4901-988c-af9753f6df6b","Type":"ContainerDied","Data":"a20cb67ac71f9d6c68e18f07875385b07a4612276eb00bef0db1baca44d0449f"}
Oct 29 22:30:05.407543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:05.407525    4860 scope.go:122] "RemoveContainer" containerID="a20cb67ac71f9d6c68e18f07875385b07a4612276eb00bef0db1baca44d0449f"
Oct 29 22:30:05.407729 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:05.407712    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw_odh-ai-gateway-infra(f7843bdf-8d86-4901-988c-af9753f6df6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b"
Oct 29 22:30:05.418549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:05.418531    4860 scope.go:122] "RemoveContainer" containerID="4564948b2e91c6a80e8ac9aca34414dad4b720d077c5978bb85f1fc43a03945c"
Oct 29 22:30:06.416444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:06.416405    4860 scope.go:122] "RemoveContainer" containerID="ec1fd491629bc8cbc210553d6109fcd8e9e19858ef8b72d759eaf203a7b415a9"
Oct 29 22:30:06.416897 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:06.416667    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9_odh-ai-gateway-infra(647ac886-ea39-4651-9903-a242645e1dcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" podUID="647ac886-ea39-4651-9903-a242645e1dcd"
Oct 29 22:30:06.418079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:06.418052    4860 scope.go:122] "RemoveContainer" containerID="a20cb67ac71f9d6c68e18f07875385b07a4612276eb00bef0db1baca44d0449f"
Oct 29 22:30:06.418253 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:06.418236    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw_odh-ai-gateway-infra(f7843bdf-8d86-4901-988c-af9753f6df6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b"
Oct 29 22:30:06.560473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:06.560442    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:07.073368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.073337    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"]
Oct 29 22:30:07.084830 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.084800    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.101210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.101177    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"]
Oct 29 22:30:07.107261 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.107233    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-43b37d\"/\"distinct-2-w3-43b37d-kserve-self-signed-certs\""
Oct 29 22:30:07.151544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.151515    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:07.195654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.195629    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tmp-dir\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.195772 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.195662    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-dshm\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.195874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.195852    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpjvd\" (UniqueName: \"kubernetes.io/projected/62e206a5-c7f6-4865-98bd-ca3a68f5312a-kube-api-access-tpjvd\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.195941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.195896    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tls-certs\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.196022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.196000    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-home\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.196121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.196105    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-model-cache\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.297513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.297474    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-home\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.297697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.297579    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-model-cache\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.297787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.297741    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tmp-dir\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.297787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.297774    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-dshm\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.297897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.297830    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpjvd\" (UniqueName: \"kubernetes.io/projected/62e206a5-c7f6-4865-98bd-ca3a68f5312a-kube-api-access-tpjvd\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.297897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.297865    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tls-certs\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.303586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.298818    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tmp-dir\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.303586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.299088    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-home\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.303586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.299289    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-model-cache\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.303586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.301352    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tls-certs\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.303586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.301478    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-dshm\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.323343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.322982    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpjvd\" (UniqueName: \"kubernetes.io/projected/62e206a5-c7f6-4865-98bd-ca3a68f5312a-kube-api-access-tpjvd\") pod \"distinct-2-w3-43b37d-kserve-587895fc96-tmhsw\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") " pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.409866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.409790    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:07.707571 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:30:07.707532    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62e206a5_c7f6_4865_98bd_ca3a68f5312a.slice/crio-805a631c2d295701b968303c816d02e5e00424d69cd838234e60c9aa4c9cb351 WatchSource:0}: Error finding container 805a631c2d295701b968303c816d02e5e00424d69cd838234e60c9aa4c9cb351: Status 404 returned error can't find the container with id 805a631c2d295701b968303c816d02e5e00424d69cd838234e60c9aa4c9cb351
Oct 29 22:30:07.710840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:07.710814    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"]
Oct 29 22:30:08.338613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:08.338578    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:08.436625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:08.436590    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" event={"ID":"62e206a5-c7f6-4865-98bd-ca3a68f5312a","Type":"ContainerStarted","Data":"19a2ffaea059c86af3276569651c889b91a589a77ab2640a4ef02875b14d917f"}
Oct 29 22:30:08.436625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:08.436636    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" event={"ID":"62e206a5-c7f6-4865-98bd-ca3a68f5312a","Type":"ContainerStarted","Data":"805a631c2d295701b968303c816d02e5e00424d69cd838234e60c9aa4c9cb351"}
Oct 29 22:30:08.480103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:08.480064    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podStartSLOduration=1.4800513450000001 podStartE2EDuration="1.480051345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:08.479456734 +0000 UTC m=+950.504232894" watchObservedRunningTime="2026-10-29 22:30:08.480051345 +0000 UTC m=+950.504827479"
Oct 29 22:30:09.410151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:09.410121    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:10.200273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:10.200240    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:10.424598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:10.424559    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:11.920057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:11.920021    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:12.203411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:12.203383    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:12.215509 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:12.215471    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:12.479573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:12.479501    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:30:13.540249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:13.540218    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:15.104561 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:15.104527    4860 scope.go:122] "RemoveContainer" containerID="55b1d44caab9ad33799b774713fab439f849ec6adad23b1b55b2b55bad55f54e"
Oct 29 22:30:15.253036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:15.253005    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:15.491956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:15.491926    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b" event={"ID":"b8a7d489-a427-427d-999e-0e0268c79457","Type":"ContainerStarted","Data":"59c586227e38e1bb3447c0d04f59db4948dfc003f2f7d1ce1c3d2484156118a9"}
Oct 29 22:30:15.533106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:15.533064    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b" podStartSLOduration=15.533051805 podStartE2EDuration="15.533051805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:15.532250326 +0000 UTC m=+957.557026459" watchObservedRunningTime="2026-10-29 22:30:15.533051805 +0000 UTC m=+957.557827936"
Oct 29 22:30:16.104569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:16.104534    4860 scope.go:122] "RemoveContainer" containerID="ec1fd491629bc8cbc210553d6109fcd8e9e19858ef8b72d759eaf203a7b415a9"
Oct 29 22:30:16.104992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:16.104784    4860 scope.go:122] "RemoveContainer" containerID="a20cb67ac71f9d6c68e18f07875385b07a4612276eb00bef0db1baca44d0449f"
Oct 29 22:30:16.500731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:16.500705    4860 generic.go:356] "Generic (PLEG): container finished" podID="b8a7d489-a427-427d-999e-0e0268c79457" containerID="59c586227e38e1bb3447c0d04f59db4948dfc003f2f7d1ce1c3d2484156118a9" exitCode=60
Oct 29 22:30:16.500849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:16.500773    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b" event={"ID":"b8a7d489-a427-427d-999e-0e0268c79457","Type":"ContainerDied","Data":"59c586227e38e1bb3447c0d04f59db4948dfc003f2f7d1ce1c3d2484156118a9"}
Oct 29 22:30:16.500849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:16.500828    4860 scope.go:122] "RemoveContainer" containerID="55b1d44caab9ad33799b774713fab439f849ec6adad23b1b55b2b55bad55f54e"
Oct 29 22:30:16.501132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:16.501116    4860 scope.go:122] "RemoveContainer" containerID="59c586227e38e1bb3447c0d04f59db4948dfc003f2f7d1ce1c3d2484156118a9"
Oct 29 22:30:16.501444 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:16.501420    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b_odh-ai-gateway-infra(b8a7d489-a427-427d-999e-0e0268c79457)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b" podUID="b8a7d489-a427-427d-999e-0e0268c79457"
Oct 29 22:30:16.502822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:16.502802    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" event={"ID":"647ac886-ea39-4651-9903-a242645e1dcd","Type":"ContainerStarted","Data":"4739ea77b4d51a2cae1ac2809d8c373885395b064420f3a315a2c45dd2c595f6"}
Oct 29 22:30:16.504559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:16.504512    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw" event={"ID":"f7843bdf-8d86-4901-988c-af9753f6df6b","Type":"ContainerStarted","Data":"6a3a3fe891c94e8ce003ebe05c15bb7175ef75a212e6915242570aec5284d13b"}
Oct 29 22:30:16.544994 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:16.544966    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"]
Oct 29 22:30:16.927862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:16.927826    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:17.410012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.409981    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:17.424066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.424037    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:17.519344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.519293    4860 generic.go:356] "Generic (PLEG): container finished" podID="647ac886-ea39-4651-9903-a242645e1dcd" containerID="4739ea77b4d51a2cae1ac2809d8c373885395b064420f3a315a2c45dd2c595f6" exitCode=60
Oct 29 22:30:17.519515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.519454    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" event={"ID":"647ac886-ea39-4651-9903-a242645e1dcd","Type":"ContainerDied","Data":"4739ea77b4d51a2cae1ac2809d8c373885395b064420f3a315a2c45dd2c595f6"}
Oct 29 22:30:17.519573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.519517    4860 scope.go:122] "RemoveContainer" containerID="ec1fd491629bc8cbc210553d6109fcd8e9e19858ef8b72d759eaf203a7b415a9"
Oct 29 22:30:17.519903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.519876    4860 scope.go:122] "RemoveContainer" containerID="4739ea77b4d51a2cae1ac2809d8c373885395b064420f3a315a2c45dd2c595f6"
Oct 29 22:30:17.520165 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:17.520144    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9_odh-ai-gateway-infra(647ac886-ea39-4651-9903-a242645e1dcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" podUID="647ac886-ea39-4651-9903-a242645e1dcd"
Oct 29 22:30:17.523256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.523228    4860 generic.go:356] "Generic (PLEG): container finished" podID="f7843bdf-8d86-4901-988c-af9753f6df6b" containerID="6a3a3fe891c94e8ce003ebe05c15bb7175ef75a212e6915242570aec5284d13b" exitCode=60
Oct 29 22:30:17.523444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.523421    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw" event={"ID":"f7843bdf-8d86-4901-988c-af9753f6df6b","Type":"ContainerDied","Data":"6a3a3fe891c94e8ce003ebe05c15bb7175ef75a212e6915242570aec5284d13b"}
Oct 29 22:30:17.523918 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.523899    4860 scope.go:122] "RemoveContainer" containerID="6a3a3fe891c94e8ce003ebe05c15bb7175ef75a212e6915242570aec5284d13b"
Oct 29 22:30:17.524346 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:17.524307    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw_odh-ai-gateway-infra(f7843bdf-8d86-4901-988c-af9753f6df6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b"
Oct 29 22:30:17.552403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.552368    4860 scope.go:122] "RemoveContainer" containerID="a20cb67ac71f9d6c68e18f07875385b07a4612276eb00bef0db1baca44d0449f"
Oct 29 22:30:17.553471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.553448    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:30:17.613170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.613138    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"]
Oct 29 22:30:17.668577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.668537    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"]
Oct 29 22:30:17.821119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.821000    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"
Oct 29 22:30:17.828246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.828213    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a7d489-a427-427d-999e-0e0268c79457-kube-api-access-mcstq\" (UniqueName: \"kubernetes.io/projected/b8a7d489-a427-427d-999e-0e0268c79457-kube-api-access-mcstq\") pod \"b8a7d489-a427-427d-999e-0e0268c79457\" (UID: \"b8a7d489-a427-427d-999e-0e0268c79457\") "
Oct 29 22:30:17.828941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.828479    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a7d489-a427-427d-999e-0e0268c79457-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8a7d489-a427-427d-999e-0e0268c79457-maas-api-service-ca\") pod \"b8a7d489-a427-427d-999e-0e0268c79457\" (UID: \"b8a7d489-a427-427d-999e-0e0268c79457\") "
Oct 29 22:30:17.829272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.829247    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a7d489-a427-427d-999e-0e0268c79457-maas-api-service-ca" pod "b8a7d489-a427-427d-999e-0e0268c79457" (UID: "b8a7d489-a427-427d-999e-0e0268c79457"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:17.836448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.831403    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a7d489-a427-427d-999e-0e0268c79457-kube-api-access-mcstq" pod "b8a7d489-a427-427d-999e-0e0268c79457" (UID: "b8a7d489-a427-427d-999e-0e0268c79457"). InnerVolumeSpecName "kube-api-access-mcstq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:17.932344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.932285    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8a7d489-a427-427d-999e-0e0268c79457-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:17.932344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:17.932348    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcstq\" (UniqueName: \"kubernetes.io/projected/b8a7d489-a427-427d-999e-0e0268c79457-kube-api-access-mcstq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:18.110391 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:18.103140    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:30:18.110391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:18.105028    4860 scope.go:122] "RemoveContainer" containerID="ce486ff2b9a60928bad296b95580299539724dca6d429de1bed778cc842a3845"
Oct 29 22:30:18.726349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:18.724640    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b" event={"ID":"b8a7d489-a427-427d-999e-0e0268c79457","Type":"ContainerDied","Data":"b30c96b66cbf87c40f43f7d449a48d2639d94fe6b15402cb38920a4b9d028edc"}
Oct 29 22:30:18.726349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:18.724787    4860 scope.go:122] "RemoveContainer" containerID="59c586227e38e1bb3447c0d04f59db4948dfc003f2f7d1ce1c3d2484156118a9"
Oct 29 22:30:18.726349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:18.724969    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"
Oct 29 22:30:18.805225 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:18.805139    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:18.944954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:18.944930    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"]
Oct 29 22:30:18.952873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:18.952839    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888550-7z97b"]
Oct 29 22:30:19.113364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.106081    4860 scope.go:122] "RemoveContainer" containerID="c386ea7da6e5ebff0cc157b820ba2f7fb5f6de7ffe57565897dc1f3c350a111f"
Oct 29 22:30:19.625541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.625510    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"
Oct 29 22:30:19.627090 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.627061    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7843bdf-8d86-4901-988c-af9753f6df6b-kube-api-access-l6qdf\" (UniqueName: \"kubernetes.io/projected/f7843bdf-8d86-4901-988c-af9753f6df6b-kube-api-access-l6qdf\") pod \"f7843bdf-8d86-4901-988c-af9753f6df6b\" (UID: \"f7843bdf-8d86-4901-988c-af9753f6df6b\") "
Oct 29 22:30:19.658657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.658629    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7843bdf-8d86-4901-988c-af9753f6df6b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7843bdf-8d86-4901-988c-af9753f6df6b-maas-api-service-ca\") pod \"f7843bdf-8d86-4901-988c-af9753f6df6b\" (UID: \"f7843bdf-8d86-4901-988c-af9753f6df6b\") "
Oct 29 22:30:19.659698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.659094    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7843bdf-8d86-4901-988c-af9753f6df6b-maas-api-service-ca" pod "f7843bdf-8d86-4901-988c-af9753f6df6b" (UID: "f7843bdf-8d86-4901-988c-af9753f6df6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:19.698886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.698105    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7843bdf-8d86-4901-988c-af9753f6df6b-kube-api-access-l6qdf" pod "f7843bdf-8d86-4901-988c-af9753f6df6b" (UID: "f7843bdf-8d86-4901-988c-af9753f6df6b"). InnerVolumeSpecName "kube-api-access-l6qdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:19.705681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.705369    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"
Oct 29 22:30:19.818116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.787049    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647ac886-ea39-4651-9903-a242645e1dcd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647ac886-ea39-4651-9903-a242645e1dcd-maas-api-service-ca\") pod \"647ac886-ea39-4651-9903-a242645e1dcd\" (UID: \"647ac886-ea39-4651-9903-a242645e1dcd\") "
Oct 29 22:30:19.818116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.787241    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647ac886-ea39-4651-9903-a242645e1dcd-kube-api-access-9qnz4\" (UniqueName: \"kubernetes.io/projected/647ac886-ea39-4651-9903-a242645e1dcd-kube-api-access-9qnz4\") pod \"647ac886-ea39-4651-9903-a242645e1dcd\" (UID: \"647ac886-ea39-4651-9903-a242645e1dcd\") "
Oct 29 22:30:19.818116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.787983    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7843bdf-8d86-4901-988c-af9753f6df6b-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:19.818116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.788009    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6qdf\" (UniqueName: \"kubernetes.io/projected/f7843bdf-8d86-4901-988c-af9753f6df6b-kube-api-access-l6qdf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:19.854471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.847435    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647ac886-ea39-4651-9903-a242645e1dcd-maas-api-service-ca" pod "647ac886-ea39-4651-9903-a242645e1dcd" (UID: "647ac886-ea39-4651-9903-a242645e1dcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:19.874398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.871534    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw" event={"ID":"f7843bdf-8d86-4901-988c-af9753f6df6b","Type":"ContainerDied","Data":"3d5e3ec70a714a39cc3439f0b2a0b60d540a1f2a07a0ce33898cba8d5982e3e7"}
Oct 29 22:30:19.874398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.871600    4860 scope.go:122] "RemoveContainer" containerID="6a3a3fe891c94e8ce003ebe05c15bb7175ef75a212e6915242570aec5284d13b"
Oct 29 22:30:19.874398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.871686    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"
Oct 29 22:30:19.905021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.904996    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647ac886-ea39-4651-9903-a242645e1dcd-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:19.980447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:19.965082    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647ac886-ea39-4651-9903-a242645e1dcd-kube-api-access-9qnz4" pod "647ac886-ea39-4651-9903-a242645e1dcd" (UID: "647ac886-ea39-4651-9903-a242645e1dcd"). InnerVolumeSpecName "kube-api-access-9qnz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:20.000956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.000908    4860 generic.go:356] "Generic (PLEG): container finished" podID="0744a866-252d-4670-97aa-b11148022f06" containerID="0ab3b7c38dc1f1d2618579ff6c7036062d596f722599104e7626843a3588a386" exitCode=60
Oct 29 22:30:20.001121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.001041    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr" event={"ID":"0744a866-252d-4670-97aa-b11148022f06","Type":"ContainerDied","Data":"0ab3b7c38dc1f1d2618579ff6c7036062d596f722599104e7626843a3588a386"}
Oct 29 22:30:20.005368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.003579    4860 scope.go:122] "RemoveContainer" containerID="0ab3b7c38dc1f1d2618579ff6c7036062d596f722599104e7626843a3588a386"
Oct 29 22:30:20.005368 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:20.003958    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr_odh-ai-gateway-infra(0744a866-252d-4670-97aa-b11148022f06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr" podUID="0744a866-252d-4670-97aa-b11148022f06"
Oct 29 22:30:20.006553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.006529    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qnz4\" (UniqueName: \"kubernetes.io/projected/647ac886-ea39-4651-9903-a242645e1dcd-kube-api-access-9qnz4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:20.032345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.027072    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"]
Oct 29 22:30:20.050551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.047942    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9" event={"ID":"647ac886-ea39-4651-9903-a242645e1dcd","Type":"ContainerDied","Data":"fac7d072ca0bd5f92ea2d2fe0ddb9952891e7ad6d8c2bb8ab9ab0f4e4ebbda87"}
Oct 29 22:30:20.050551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.048057    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"
Oct 29 22:30:20.054607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.053054    4860 scope.go:122] "RemoveContainer" containerID="ce486ff2b9a60928bad296b95580299539724dca6d429de1bed778cc842a3845"
Oct 29 22:30:20.099377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.094034    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888550-flmtw"]
Oct 29 22:30:20.140707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.140682    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a7d489-a427-427d-999e-0e0268c79457" path="/var/lib/kubelet/pods/b8a7d489-a427-427d-999e-0e0268c79457/volumes"
Oct 29 22:30:20.141501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.141481    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b" path="/var/lib/kubelet/pods/f7843bdf-8d86-4901-988c-af9753f6df6b/volumes"
Oct 29 22:30:20.155002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.154984    4860 scope.go:122] "RemoveContainer" containerID="4739ea77b4d51a2cae1ac2809d8c373885395b064420f3a315a2c45dd2c595f6"
Oct 29 22:30:20.171188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.171160    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"]
Oct 29 22:30:20.175424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.175402    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-43b37d-29888550-hnhq9"]
Oct 29 22:30:20.808990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:20.808954    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:21.057725 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:21.057691    4860 generic.go:356] "Generic (PLEG): container finished" podID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" containerID="a350cd290f088b4f0ee1f225fc9c99ebc483228b8fd03d6343802385ef6a712b" exitCode=60
Oct 29 22:30:21.058140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:21.057765    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54" event={"ID":"e5f90084-0afe-430f-a9a9-f7f0775e02f2","Type":"ContainerDied","Data":"a350cd290f088b4f0ee1f225fc9c99ebc483228b8fd03d6343802385ef6a712b"}
Oct 29 22:30:21.058140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:21.057821    4860 scope.go:122] "RemoveContainer" containerID="c386ea7da6e5ebff0cc157b820ba2f7fb5f6de7ffe57565897dc1f3c350a111f"
Oct 29 22:30:21.058692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:21.058668    4860 scope.go:122] "RemoveContainer" containerID="a350cd290f088b4f0ee1f225fc9c99ebc483228b8fd03d6343802385ef6a712b"
Oct 29 22:30:21.059369 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:21.058975    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54_odh-ai-gateway-infra(e5f90084-0afe-430f-a9a9-f7f0775e02f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2"
Oct 29 22:30:21.063952 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:21.063932    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"]
Oct 29 22:30:21.217506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:21.217483    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"
Oct 29 22:30:21.330993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:21.330909    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0744a866-252d-4670-97aa-b11148022f06-kube-api-access-5jbjc\" (UniqueName: \"kubernetes.io/projected/0744a866-252d-4670-97aa-b11148022f06-kube-api-access-5jbjc\") pod \"0744a866-252d-4670-97aa-b11148022f06\" (UID: \"0744a866-252d-4670-97aa-b11148022f06\") "
Oct 29 22:30:21.331136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:21.331059    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0744a866-252d-4670-97aa-b11148022f06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0744a866-252d-4670-97aa-b11148022f06-maas-api-service-ca\") pod \"0744a866-252d-4670-97aa-b11148022f06\" (UID: \"0744a866-252d-4670-97aa-b11148022f06\") "
Oct 29 22:30:21.331415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:21.331392    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0744a866-252d-4670-97aa-b11148022f06-maas-api-service-ca" pod "0744a866-252d-4670-97aa-b11148022f06" (UID: "0744a866-252d-4670-97aa-b11148022f06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:21.331556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:21.331546    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0744a866-252d-4670-97aa-b11148022f06-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:21.333010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:21.332983    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0744a866-252d-4670-97aa-b11148022f06-kube-api-access-5jbjc" pod "0744a866-252d-4670-97aa-b11148022f06" (UID: "0744a866-252d-4670-97aa-b11148022f06"). InnerVolumeSpecName "kube-api-access-5jbjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:21.432303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:21.432275    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jbjc\" (UniqueName: \"kubernetes.io/projected/0744a866-252d-4670-97aa-b11148022f06-kube-api-access-5jbjc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:22.073578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.073548    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"
Oct 29 22:30:22.073578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.073572    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr" event={"ID":"0744a866-252d-4670-97aa-b11148022f06","Type":"ContainerDied","Data":"a9d995273cb0f02990c8a837a032202e16e118e71ba37da1c1fb6fd99d9a8c69"}
Oct 29 22:30:22.074102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.073666    4860 scope.go:122] "RemoveContainer" containerID="0ab3b7c38dc1f1d2618579ff6c7036062d596f722599104e7626843a3588a386"
Oct 29 22:30:22.113963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.113934    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647ac886-ea39-4651-9903-a242645e1dcd" path="/var/lib/kubelet/pods/647ac886-ea39-4651-9903-a242645e1dcd/volumes"
Oct 29 22:30:22.114341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.114312    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"]
Oct 29 22:30:22.131692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.131664    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"]
Oct 29 22:30:22.140102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.140071    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888550-k9ctr"]
Oct 29 22:30:22.283488 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.283464    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"
Oct 29 22:30:22.407522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.407448    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:22.440851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.440827    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f90084-0afe-430f-a9a9-f7f0775e02f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f90084-0afe-430f-a9a9-f7f0775e02f2-maas-api-service-ca\") pod \"e5f90084-0afe-430f-a9a9-f7f0775e02f2\" (UID: \"e5f90084-0afe-430f-a9a9-f7f0775e02f2\") "
Oct 29 22:30:22.440966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.440910    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f90084-0afe-430f-a9a9-f7f0775e02f2-kube-api-access-c554w\" (UniqueName: \"kubernetes.io/projected/e5f90084-0afe-430f-a9a9-f7f0775e02f2-kube-api-access-c554w\") pod \"e5f90084-0afe-430f-a9a9-f7f0775e02f2\" (UID: \"e5f90084-0afe-430f-a9a9-f7f0775e02f2\") "
Oct 29 22:30:22.441164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.441143    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f90084-0afe-430f-a9a9-f7f0775e02f2-maas-api-service-ca" pod "e5f90084-0afe-430f-a9a9-f7f0775e02f2" (UID: "e5f90084-0afe-430f-a9a9-f7f0775e02f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:22.441434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.441411    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f90084-0afe-430f-a9a9-f7f0775e02f2-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:22.443119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.443098    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f90084-0afe-430f-a9a9-f7f0775e02f2-kube-api-access-c554w" pod "e5f90084-0afe-430f-a9a9-f7f0775e02f2" (UID: "e5f90084-0afe-430f-a9a9-f7f0775e02f2"). InnerVolumeSpecName "kube-api-access-c554w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:22.543257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:22.543220    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c554w\" (UniqueName: \"kubernetes.io/projected/e5f90084-0afe-430f-a9a9-f7f0775e02f2-kube-api-access-c554w\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:23.083138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:23.083109    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"
Oct 29 22:30:23.083628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:23.083139    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54" event={"ID":"e5f90084-0afe-430f-a9a9-f7f0775e02f2","Type":"ContainerDied","Data":"3a0a6870c91659289fbd445f03a1417c83c007f4da54e764ecbe454a9a9ece25"}
Oct 29 22:30:23.083628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:23.083199    4860 scope.go:122] "RemoveContainer" containerID="a350cd290f088b4f0ee1f225fc9c99ebc483228b8fd03d6343802385ef6a712b"
Oct 29 22:30:23.139579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:23.139554    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"]
Oct 29 22:30:23.156347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:23.154087    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888550-lfk54"]
Oct 29 22:30:24.111842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:24.111810    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0744a866-252d-4670-97aa-b11148022f06" path="/var/lib/kubelet/pods/0744a866-252d-4670-97aa-b11148022f06/volumes"
Oct 29 22:30:24.112216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:24.112202    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" path="/var/lib/kubelet/pods/e5f90084-0afe-430f-a9a9-f7f0775e02f2/volumes"
Oct 29 22:30:24.164500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:24.164475    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:25.233544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.233513    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"]
Oct 29 22:30:25.234039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234025    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" containerName="cleanup"
Oct 29 22:30:25.234039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234038    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234053    4860 cpu_manager.go:414] "containerMap: removing container" podUID="647ac886-ea39-4651-9903-a242645e1dcd" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234058    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647ac886-ea39-4651-9903-a242645e1dcd" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234071    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234076    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234083    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0744a866-252d-4670-97aa-b11148022f06" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234088    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0744a866-252d-4670-97aa-b11148022f06" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234096    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234100    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234112    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234118    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234124    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b8a7d489-a427-427d-999e-0e0268c79457" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234128    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a7d489-a427-427d-999e-0e0268c79457" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234134    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0744a866-252d-4670-97aa-b11148022f06" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234138    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0744a866-252d-4670-97aa-b11148022f06" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234146    4860 cpu_manager.go:414] "containerMap: removing container" podUID="647ac886-ea39-4651-9903-a242645e1dcd" containerName="cleanup"
Oct 29 22:30:25.234146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234150    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647ac886-ea39-4651-9903-a242645e1dcd" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234175    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b8a7d489-a427-427d-999e-0e0268c79457" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234180    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a7d489-a427-427d-999e-0e0268c79457" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234276    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234285    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234291    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0744a866-252d-4670-97aa-b11148022f06" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234296    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234305    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234309    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a7d489-a427-427d-999e-0e0268c79457" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234314    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0744a866-252d-4670-97aa-b11148022f06" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234333    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="647ac886-ea39-4651-9903-a242645e1dcd" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234347    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234357    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a7d489-a427-427d-999e-0e0268c79457" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234365    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a7d489-a427-427d-999e-0e0268c79457" containerName="cleanup"
Oct 29 22:30:25.234962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.234373    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="647ac886-ea39-4651-9903-a242645e1dcd" containerName="cleanup"
Oct 29 22:30:25.239244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.239224    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.257230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.257207    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"]
Oct 29 22:30:25.264583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.264563    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2ba10a\"/\"distinct-2-w2-2ba10a-kserve-self-signed-certs\""
Oct 29 22:30:25.314823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.314794    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:25.372723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.372700    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/482698c2-d44a-4564-b588-d402fd9cac23-tls-certs\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.372838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.372737    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-tmp-dir\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.372838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.372760    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-model-cache\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.372918 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.372895    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6f7s\" (UniqueName: \"kubernetes.io/projected/482698c2-d44a-4564-b588-d402fd9cac23-kube-api-access-t6f7s\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.372964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.372952    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-home\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.372996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.372986    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-dshm\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.474225 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.474185    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6f7s\" (UniqueName: \"kubernetes.io/projected/482698c2-d44a-4564-b588-d402fd9cac23-kube-api-access-t6f7s\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.474447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.474251    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-home\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.474447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.474310    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-dshm\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.474541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.474521    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/482698c2-d44a-4564-b588-d402fd9cac23-tls-certs\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.474667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.474572    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-tmp-dir\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.474667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.474621    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-model-cache\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.475351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.475295    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-home\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.475692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.475395    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-model-cache\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.475692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.475410    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-tmp-dir\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.477166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.477145    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-dshm\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.477622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.477595    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/482698c2-d44a-4564-b588-d402fd9cac23-tls-certs\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.495096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.495038    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6f7s\" (UniqueName: \"kubernetes.io/projected/482698c2-d44a-4564-b588-d402fd9cac23-kube-api-access-t6f7s\") pod \"distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") " pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.561861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.561825    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:25.862711 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:30:25.862676    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod482698c2_d44a_4564_b588_d402fd9cac23.slice/crio-877f1b7a1e958a2dcccf8177a006d64cfc7b1a524cff4455213301740538435e WatchSource:0}: Error finding container 877f1b7a1e958a2dcccf8177a006d64cfc7b1a524cff4455213301740538435e: Status 404 returned error can't find the container with id 877f1b7a1e958a2dcccf8177a006d64cfc7b1a524cff4455213301740538435e
Oct 29 22:30:25.865966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.865943    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"]
Oct 29 22:30:25.883457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:25.883432    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:26.113050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:26.112975    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" event={"ID":"482698c2-d44a-4564-b588-d402fd9cac23","Type":"ContainerStarted","Data":"c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e"}
Oct 29 22:30:26.113050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:26.113013    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" event={"ID":"482698c2-d44a-4564-b588-d402fd9cac23","Type":"ContainerStarted","Data":"877f1b7a1e958a2dcccf8177a006d64cfc7b1a524cff4455213301740538435e"}
Oct 29 22:30:26.154910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:26.154867    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podStartSLOduration=1.154853287 podStartE2EDuration="1.154853287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:26.153778397 +0000 UTC m=+968.178554539" watchObservedRunningTime="2026-10-29 22:30:26.154853287 +0000 UTC m=+968.179629447"
Oct 29 22:30:26.562783 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:26.562754    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:27.569217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:27.569187    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:28.239831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.239801    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"]
Oct 29 22:30:28.240363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240348    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b8a7d489-a427-427d-999e-0e0268c79457" containerName="cleanup"
Oct 29 22:30:28.240363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240362    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a7d489-a427-427d-999e-0e0268c79457" containerName="cleanup"
Oct 29 22:30:28.240438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240377    4860 cpu_manager.go:414] "containerMap: removing container" podUID="647ac886-ea39-4651-9903-a242645e1dcd" containerName="cleanup"
Oct 29 22:30:28.240438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240383    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647ac886-ea39-4651-9903-a242645e1dcd" containerName="cleanup"
Oct 29 22:30:28.240438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240399    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" containerName="cleanup"
Oct 29 22:30:28.240438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240404    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" containerName="cleanup"
Oct 29 22:30:28.240438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240415    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b" containerName="cleanup"
Oct 29 22:30:28.240438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240420    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7843bdf-8d86-4901-988c-af9753f6df6b" containerName="cleanup"
Oct 29 22:30:28.240438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240428    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0744a866-252d-4670-97aa-b11148022f06" containerName="cleanup"
Oct 29 22:30:28.240438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240432    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0744a866-252d-4670-97aa-b11148022f06" containerName="cleanup"
Oct 29 22:30:28.240713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240519    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="647ac886-ea39-4651-9903-a242645e1dcd" containerName="cleanup"
Oct 29 22:30:28.240713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240528    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f90084-0afe-430f-a9a9-f7f0775e02f2" containerName="cleanup"
Oct 29 22:30:28.240713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.240537    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0744a866-252d-4670-97aa-b11148022f06" containerName="cleanup"
Oct 29 22:30:28.243734 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.243718    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:28.261297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.261273    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"]
Oct 29 22:30:28.270337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.270296    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:30:28.270447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.270316    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:30:28.270447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.270319    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:30:28.273054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.273037    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:30:28.408263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.408235    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8959cc98-fxftk\" (UID: \"49f69917-9bbe-452b-8b61-35bd89d82705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:28.408409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.408279    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8959cc98-fxftk\" (UID: \"49f69917-9bbe-452b-8b61-35bd89d82705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:28.408409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.408301    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9k7g\" (UniqueName: \"kubernetes.io/projected/49f69917-9bbe-452b-8b61-35bd89d82705-kube-api-access-h9k7g\") pod \"maas-api-revoke-sub-8959cc98-fxftk\" (UID: \"49f69917-9bbe-452b-8b61-35bd89d82705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:28.509048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.508964    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8959cc98-fxftk\" (UID: \"49f69917-9bbe-452b-8b61-35bd89d82705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:28.509048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.509017    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8959cc98-fxftk\" (UID: \"49f69917-9bbe-452b-8b61-35bd89d82705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:28.509048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.509038    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9k7g\" (UniqueName: \"kubernetes.io/projected/49f69917-9bbe-452b-8b61-35bd89d82705-kube-api-access-h9k7g\") pod \"maas-api-revoke-sub-8959cc98-fxftk\" (UID: \"49f69917-9bbe-452b-8b61-35bd89d82705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:28.509727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.509701    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8959cc98-fxftk\" (UID: \"49f69917-9bbe-452b-8b61-35bd89d82705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:28.511635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.511610    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8959cc98-fxftk\" (UID: \"49f69917-9bbe-452b-8b61-35bd89d82705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:28.529872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.529846    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9k7g\" (UniqueName: \"kubernetes.io/projected/49f69917-9bbe-452b-8b61-35bd89d82705-kube-api-access-h9k7g\") pod \"maas-api-revoke-sub-8959cc98-fxftk\" (UID: \"49f69917-9bbe-452b-8b61-35bd89d82705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:28.558589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.558566    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:28.858445 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:30:28.858418    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f69917_9bbe_452b_8b61_35bd89d82705.slice/crio-d766163b88387bf4684c6417fc593c909cd95ecd4ff235956cbe61c23a708d0a WatchSource:0}: Error finding container d766163b88387bf4684c6417fc593c909cd95ecd4ff235956cbe61c23a708d0a: Status 404 returned error can't find the container with id d766163b88387bf4684c6417fc593c909cd95ecd4ff235956cbe61c23a708d0a
Oct 29 22:30:28.862206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:28.862182    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"]
Oct 29 22:30:29.135957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:29.135887    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk" event={"ID":"49f69917-9bbe-452b-8b61-35bd89d82705","Type":"ContainerStarted","Data":"d766163b88387bf4684c6417fc593c909cd95ecd4ff235956cbe61c23a708d0a"}
Oct 29 22:30:29.169653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:29.169626    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:30.146076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:30.145988    4860 generic.go:356] "Generic (PLEG): container finished" podID="49f69917-9bbe-452b-8b61-35bd89d82705" containerID="8b03a95830a4b00e14fe2fe21b67224ab8d18cab2578749de2f88f830ef01630" exitCode=0
Oct 29 22:30:30.146531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:30.146067    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk" event={"ID":"49f69917-9bbe-452b-8b61-35bd89d82705","Type":"ContainerDied","Data":"8b03a95830a4b00e14fe2fe21b67224ab8d18cab2578749de2f88f830ef01630"}
Oct 29 22:30:30.930727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:30.930696    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:31.300717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:31.300693    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:31.347973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:31.347941    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-xks-ca\") pod \"49f69917-9bbe-452b-8b61-35bd89d82705\" (UID: \"49f69917-9bbe-452b-8b61-35bd89d82705\") "
Oct 29 22:30:31.348137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:31.348011    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-openshift-ca\") pod \"49f69917-9bbe-452b-8b61-35bd89d82705\" (UID: \"49f69917-9bbe-452b-8b61-35bd89d82705\") "
Oct 29 22:30:31.348207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:31.348185    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f69917-9bbe-452b-8b61-35bd89d82705-kube-api-access-h9k7g\" (UniqueName: \"kubernetes.io/projected/49f69917-9bbe-452b-8b61-35bd89d82705-kube-api-access-h9k7g\") pod \"49f69917-9bbe-452b-8b61-35bd89d82705\" (UID: \"49f69917-9bbe-452b-8b61-35bd89d82705\") "
Oct 29 22:30:31.348753 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:31.348717    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-openshift-ca" pod "49f69917-9bbe-452b-8b61-35bd89d82705" (UID: "49f69917-9bbe-452b-8b61-35bd89d82705"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:31.351243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:31.351200    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f69917-9bbe-452b-8b61-35bd89d82705-kube-api-access-h9k7g" pod "49f69917-9bbe-452b-8b61-35bd89d82705" (UID: "49f69917-9bbe-452b-8b61-35bd89d82705"). InnerVolumeSpecName "kube-api-access-h9k7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:31.351902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:31.351875    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-xks-ca" pod "49f69917-9bbe-452b-8b61-35bd89d82705" (UID: "49f69917-9bbe-452b-8b61-35bd89d82705"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:30:31.449574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:31.449543    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:31.449574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:31.449572    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9k7g\" (UniqueName: \"kubernetes.io/projected/49f69917-9bbe-452b-8b61-35bd89d82705-kube-api-access-h9k7g\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:31.449741 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:31.449583    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f69917-9bbe-452b-8b61-35bd89d82705-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:32.162035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:32.162005    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"
Oct 29 22:30:32.162035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:32.162022    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk" event={"ID":"49f69917-9bbe-452b-8b61-35bd89d82705","Type":"ContainerDied","Data":"d766163b88387bf4684c6417fc593c909cd95ecd4ff235956cbe61c23a708d0a"}
Oct 29 22:30:32.162245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:32.162077    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d766163b88387bf4684c6417fc593c909cd95ecd4ff235956cbe61c23a708d0a"
Oct 29 22:30:32.634784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:32.634748    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:34.392056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:34.392013    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:35.562397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:35.562367    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:35.580259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:35.580230    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:36.027222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.027180    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:36.053818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.053787    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"]
Oct 29 22:30:36.054643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.054627    4860 cpu_manager.go:414] "containerMap: removing container" podUID="49f69917-9bbe-452b-8b61-35bd89d82705" containerName="revoke-keys"
Oct 29 22:30:36.054694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.054646    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f69917-9bbe-452b-8b61-35bd89d82705" containerName="revoke-keys"
Oct 29 22:30:36.054797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.054786    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f69917-9bbe-452b-8b61-35bd89d82705" containerName="revoke-keys"
Oct 29 22:30:36.059093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.059073    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.080928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.080905    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"]
Oct 29 22:30:36.150294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.150251    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.150533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.150394    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60189037-441a-4cb7-866c-6287bda16518-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.150533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.150447    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79nd\" (UniqueName: \"kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-kube-api-access-c79nd\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.150533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.150489    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.150533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.150516    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60189037-441a-4cb7-866c-6287bda16518-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.150752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.150579    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.150752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.150610    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.150752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.150723    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.151320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.151297    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.243926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.243899    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:30:36.252661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.252636    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.252831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.252697    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60189037-441a-4cb7-866c-6287bda16518-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.252831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.252726    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79nd\" (UniqueName: \"kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-kube-api-access-c79nd\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.252955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.252875    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.252955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.252918    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60189037-441a-4cb7-866c-6287bda16518-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.253060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.252981    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.253060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.253014    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.253182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.253092    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.253304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.253283    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.253420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.253356    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.253483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.253360    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-data\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.253483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.253442    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-credential-socket\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.253610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.253591    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-certs\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.253728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.253708    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60189037-441a-4cb7-866c-6287bda16518-istiod-ca-cert\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.255101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.255079    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-envoy\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.255390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.255372    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60189037-441a-4cb7-866c-6287bda16518-istio-podinfo\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.273151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.273125    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79nd\" (UniqueName: \"kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-kube-api-access-c79nd\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.273336 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.273304    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-istio-token\") pod \"e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") " pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.308973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.308908    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:36.380632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.380604    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:36.708992 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:30:36.708958    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60189037_441a_4cb7_866c_6287bda16518.slice/crio-3d9eda58f0fa9600175f872695043e1a0fad035993f5e18a673d1f76b54063ac WatchSource:0}: Error finding container 3d9eda58f0fa9600175f872695043e1a0fad035993f5e18a673d1f76b54063ac: Status 404 returned error can't find the container with id 3d9eda58f0fa9600175f872695043e1a0fad035993f5e18a673d1f76b54063ac
Oct 29 22:30:36.711265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.711231    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:30:36.711402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.711298    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:30:36.711402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.711361    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:30:36.712455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:36.712430    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"]
Oct 29 22:30:37.241697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.241649    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" event={"ID":"60189037-441a-4cb7-866c-6287bda16518","Type":"ContainerStarted","Data":"25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e"}
Oct 29 22:30:37.241903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.241751    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" event={"ID":"60189037-441a-4cb7-866c-6287bda16518","Type":"ContainerStarted","Data":"3d9eda58f0fa9600175f872695043e1a0fad035993f5e18a673d1f76b54063ac"}
Oct 29 22:30:37.294636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.294581    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" podStartSLOduration=1.294563393 podStartE2EDuration="1.294563393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:37.290814413 +0000 UTC m=+979.315590545" watchObservedRunningTime="2026-10-29 22:30:37.294563393 +0000 UTC m=+979.319339525"
Oct 29 22:30:37.390350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.389698    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:37.393622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.393591    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:30:37.393769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.393660    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" podUID="60189037-441a-4cb7-866c-6287bda16518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:30:37.908106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.908044    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"]
Oct 29 22:30:37.913016 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.912992    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:37.918160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.915469    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"]
Oct 29 22:30:37.929142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.929122    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:30:37.943070 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.943044    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:30:37.943167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.943078    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:30:37.943167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.943140    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:30:37.998621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.998584    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5n6t\" (UniqueName: \"kubernetes.io/projected/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-kube-api-access-t5n6t\") pod \"maas-api-revoke-sub-2339b09e-7d4s9\" (UID: \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:37.998817 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.998717    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2339b09e-7d4s9\" (UID: \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:37.998817 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:37.998760    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2339b09e-7d4s9\" (UID: \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:38.099623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:38.099594    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5n6t\" (UniqueName: \"kubernetes.io/projected/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-kube-api-access-t5n6t\") pod \"maas-api-revoke-sub-2339b09e-7d4s9\" (UID: \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:38.099818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:38.099677    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2339b09e-7d4s9\" (UID: \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:38.099818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:38.099699    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2339b09e-7d4s9\" (UID: \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:38.100422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:38.100388    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2339b09e-7d4s9\" (UID: \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:38.102127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:38.102104    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2339b09e-7d4s9\" (UID: \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:38.120977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:38.120948    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5n6t\" (UniqueName: \"kubernetes.io/projected/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-kube-api-access-t5n6t\") pod \"maas-api-revoke-sub-2339b09e-7d4s9\" (UID: \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:38.232125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:38.232092    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:38.257221 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:38.257189    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:38.391839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:38.391800    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:30:38.392099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:38.392067    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" podUID="60189037-441a-4cb7-866c-6287bda16518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:30:38.598349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:38.593166    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"]
Oct 29 22:30:39.258458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:39.258355    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9" event={"ID":"910fc8ce-5da1-4bb6-8cf5-cc120522aa59","Type":"ContainerStarted","Data":"6bba72acaf32c4d8b78550164ef1aa631750ab06622b9b1644ba78d84cd06205"}
Oct 29 22:30:39.258458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:39.258430    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9" event={"ID":"910fc8ce-5da1-4bb6-8cf5-cc120522aa59","Type":"ContainerStarted","Data":"be8c8021a8a32292afb6bae44bf7327c5641b89817fa8abfc26b08d25fc045b0"}
Oct 29 22:30:39.320726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:39.320666    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9" podStartSLOduration=2.3206525080000002 podStartE2EDuration="2.320652508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:39.317610801 +0000 UTC m=+981.342386933" watchObservedRunningTime="2026-10-29 22:30:39.320652508 +0000 UTC m=+981.345428639"
Oct 29 22:30:39.392943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:39.392900    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:30:39.393134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:39.392959    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" podUID="60189037-441a-4cb7-866c-6287bda16518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:30:40.041638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:40.041562    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:40.267157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:40.267126    4860 generic.go:356] "Generic (PLEG): container finished" podID="910fc8ce-5da1-4bb6-8cf5-cc120522aa59" containerID="6bba72acaf32c4d8b78550164ef1aa631750ab06622b9b1644ba78d84cd06205" exitCode=0
Oct 29 22:30:40.267619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:40.267203    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9" event={"ID":"910fc8ce-5da1-4bb6-8cf5-cc120522aa59","Type":"ContainerDied","Data":"6bba72acaf32c4d8b78550164ef1aa631750ab06622b9b1644ba78d84cd06205"}
Oct 29 22:30:40.395425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:40.395293    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:30:40.395675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:40.395644    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" podUID="60189037-441a-4cb7-866c-6287bda16518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:30:41.384123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:41.383681    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:30:41.384123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:41.383749    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" podUID="60189037-441a-4cb7-866c-6287bda16518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:30:41.501664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:41.501640    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:41.643436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:41.643307    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-kube-api-access-t5n6t\" (UniqueName: \"kubernetes.io/projected/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-kube-api-access-t5n6t\") pod \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\" (UID: \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\") "
Oct 29 22:30:41.643607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:41.643470    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-xks-ca\") pod \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\" (UID: \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\") "
Oct 29 22:30:41.643607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:41.643516    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-openshift-ca\") pod \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\" (UID: \"910fc8ce-5da1-4bb6-8cf5-cc120522aa59\") "
Oct 29 22:30:41.643965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:41.643932    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-openshift-ca" pod "910fc8ce-5da1-4bb6-8cf5-cc120522aa59" (UID: "910fc8ce-5da1-4bb6-8cf5-cc120522aa59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:41.645526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:41.645497    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-kube-api-access-t5n6t" pod "910fc8ce-5da1-4bb6-8cf5-cc120522aa59" (UID: "910fc8ce-5da1-4bb6-8cf5-cc120522aa59"). InnerVolumeSpecName "kube-api-access-t5n6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:41.645643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:41.645619    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-xks-ca" pod "910fc8ce-5da1-4bb6-8cf5-cc120522aa59" (UID: "910fc8ce-5da1-4bb6-8cf5-cc120522aa59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:30:41.745203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:41.745169    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5n6t\" (UniqueName: \"kubernetes.io/projected/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-kube-api-access-t5n6t\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:41.745203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:41.745199    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:41.745203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:41.745209    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/910fc8ce-5da1-4bb6-8cf5-cc120522aa59-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:42.040288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:42.040249    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:42.308133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:42.308059    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"
Oct 29 22:30:42.308133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:42.308098    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9" event={"ID":"910fc8ce-5da1-4bb6-8cf5-cc120522aa59","Type":"ContainerDied","Data":"be8c8021a8a32292afb6bae44bf7327c5641b89817fa8abfc26b08d25fc045b0"}
Oct 29 22:30:42.308344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:42.308152    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be8c8021a8a32292afb6bae44bf7327c5641b89817fa8abfc26b08d25fc045b0"
Oct 29 22:30:42.387759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:42.387723    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:30:42.388317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:42.388289    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" podUID="60189037-441a-4cb7-866c-6287bda16518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:30:43.381710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:43.381677    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:30:43.381891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:43.381734    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" podUID="60189037-441a-4cb7-866c-6287bda16518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:30:43.730943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:43.730905    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:44.385298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:44.385272    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:44.385758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:44.385737    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:44.386263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:44.386244    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:30:45.439120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:45.439094    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:47.346661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:47.346624    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:48.055518 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:30:48.055480    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:30:48.925983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:48.925948    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:50.443816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.443779    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"]
Oct 29 22:30:50.447010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.446982    4860 cpu_manager.go:414] "containerMap: removing container" podUID="910fc8ce-5da1-4bb6-8cf5-cc120522aa59" containerName="revoke-keys"
Oct 29 22:30:50.447218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.447200    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="910fc8ce-5da1-4bb6-8cf5-cc120522aa59" containerName="revoke-keys"
Oct 29 22:30:50.447579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.447561    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="910fc8ce-5da1-4bb6-8cf5-cc120522aa59" containerName="revoke-keys"
Oct 29 22:30:50.470127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.470095    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.493635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.493597    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"]
Oct 29 22:30:50.568246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.568209    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.568640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.568602    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.568819 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.568800    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528f6325-67f0-490d-b4a0-cf71638282b7-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.579618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.579523    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.579949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.579928    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.580371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.580347    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.580711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.580542    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528f6325-67f0-490d-b4a0-cf71638282b7-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.581154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.580877    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqh6\" (UniqueName: \"kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-kube-api-access-7gqh6\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.581558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.581537    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.734444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.715949    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.734444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.716080    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.734444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.716110    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528f6325-67f0-490d-b4a0-cf71638282b7-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.734444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.716174    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.734444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.716213    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.734444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.716279    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.734444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.716319    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528f6325-67f0-490d-b4a0-cf71638282b7-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.734444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.716391    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqh6\" (UniqueName: \"kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-kube-api-access-7gqh6\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.766360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.763862    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.793358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.787529    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.793358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.788024    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.793358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.788278    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.793358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.788544    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.818359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.799940    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528f6325-67f0-490d-b4a0-cf71638282b7-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.818359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.813111    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528f6325-67f0-490d-b4a0-cf71638282b7-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.917345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.907081    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqh6\" (UniqueName: \"kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-kube-api-access-7gqh6\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.917345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.915654    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:50.918108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:50.918021    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:51.152407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:51.149162    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:51.263349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:51.262343    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:51.520348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:51.508140    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:30:51.823711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:51.823680    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"]
Oct 29 22:30:51.829424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:51.829382    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:30:51.829717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:51.829690    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:30:51.829890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:51.829868    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:30:52.272524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.272492    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"]
Oct 29 22:30:52.277100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.277077    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:52.292781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.292753    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"]
Oct 29 22:30:52.302161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.301688    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:30:52.302161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.301713    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:30:52.302161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.301697    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:30:52.306779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.306761    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:30:52.353158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.353131    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8pf\" (UniqueName: \"kubernetes.io/projected/e2018efa-a97e-4104-92ff-b97be01361e4-kube-api-access-lm8pf\") pod \"maas-api-revoke-sub-21e6b7cf-9k9gq\" (UID: \"e2018efa-a97e-4104-92ff-b97be01361e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:52.353392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.353373    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e6b7cf-9k9gq\" (UID: \"e2018efa-a97e-4104-92ff-b97be01361e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:52.353458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.353411    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e6b7cf-9k9gq\" (UID: \"e2018efa-a97e-4104-92ff-b97be01361e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:52.431508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.431472    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" event={"ID":"528f6325-67f0-490d-b4a0-cf71638282b7","Type":"ContainerStarted","Data":"d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789"}
Oct 29 22:30:52.431619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.431528    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" event={"ID":"528f6325-67f0-490d-b4a0-cf71638282b7","Type":"ContainerStarted","Data":"4ff997e4af414096320c07e56686a6cfca3f4ed3214d3988ae923dea5580b7f8"}
Oct 29 22:30:52.454882 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.454855    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8pf\" (UniqueName: \"kubernetes.io/projected/e2018efa-a97e-4104-92ff-b97be01361e4-kube-api-access-lm8pf\") pod \"maas-api-revoke-sub-21e6b7cf-9k9gq\" (UID: \"e2018efa-a97e-4104-92ff-b97be01361e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:52.455010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.454982    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e6b7cf-9k9gq\" (UID: \"e2018efa-a97e-4104-92ff-b97be01361e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:52.455073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.455008    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e6b7cf-9k9gq\" (UID: \"e2018efa-a97e-4104-92ff-b97be01361e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:52.455637 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.455616    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e6b7cf-9k9gq\" (UID: \"e2018efa-a97e-4104-92ff-b97be01361e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:52.457455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.457429    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e6b7cf-9k9gq\" (UID: \"e2018efa-a97e-4104-92ff-b97be01361e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:52.475916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.475881    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8pf\" (UniqueName: \"kubernetes.io/projected/e2018efa-a97e-4104-92ff-b97be01361e4-kube-api-access-lm8pf\") pod \"maas-api-revoke-sub-21e6b7cf-9k9gq\" (UID: \"e2018efa-a97e-4104-92ff-b97be01361e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:52.476875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.476835    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" podStartSLOduration=2.476823511 podStartE2EDuration="2.476823511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:52.475192979 +0000 UTC m=+994.499969112" watchObservedRunningTime="2026-10-29 22:30:52.476823511 +0000 UTC m=+994.501599642"
Oct 29 22:30:52.592789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.592733    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:52.921314 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:30:52.921268    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2018efa_a97e_4104_92ff_b97be01361e4.slice/crio-be9e8e2b628d4f884c45dde287886c4630b1dd64c072ae302a5e6888f716b825 WatchSource:0}: Error finding container be9e8e2b628d4f884c45dde287886c4630b1dd64c072ae302a5e6888f716b825: Status 404 returned error can't find the container with id be9e8e2b628d4f884c45dde287886c4630b1dd64c072ae302a5e6888f716b825
Oct 29 22:30:52.924253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:52.924230    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"]
Oct 29 22:30:53.150526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:53.150495    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:53.152385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:53.152353    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:30:53.152507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:53.152420    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" podUID="528f6325-67f0-490d-b4a0-cf71638282b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 29 22:30:53.407315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:53.407289    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:53.445372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:53.445227    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq" event={"ID":"e2018efa-a97e-4104-92ff-b97be01361e4","Type":"ContainerStarted","Data":"6903242604c67f174c4ec5bb69594a24f3c2b47595d34080afed3cc54110919f"}
Oct 29 22:30:53.445372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:53.445295    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq" event={"ID":"e2018efa-a97e-4104-92ff-b97be01361e4","Type":"ContainerStarted","Data":"be9e8e2b628d4f884c45dde287886c4630b1dd64c072ae302a5e6888f716b825"}
Oct 29 22:30:53.497780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:53.496437    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq" podStartSLOduration=1.496420009 podStartE2EDuration="1.496420009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:53.493826013 +0000 UTC m=+995.518602172" watchObservedRunningTime="2026-10-29 22:30:53.496420009 +0000 UTC m=+995.521196140"
Oct 29 22:30:54.150738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:54.150701    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:30:54.151256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:54.151225    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" podUID="528f6325-67f0-490d-b4a0-cf71638282b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 29 22:30:54.453858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:54.453827    4860 generic.go:356] "Generic (PLEG): container finished" podID="e2018efa-a97e-4104-92ff-b97be01361e4" containerID="6903242604c67f174c4ec5bb69594a24f3c2b47595d34080afed3cc54110919f" exitCode=0
Oct 29 22:30:54.454038 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:54.453919    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq" event={"ID":"e2018efa-a97e-4104-92ff-b97be01361e4","Type":"ContainerDied","Data":"6903242604c67f174c4ec5bb69594a24f3c2b47595d34080afed3cc54110919f"}
Oct 29 22:30:55.151302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.151272    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:30:55.151707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.151336    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" podUID="528f6325-67f0-490d-b4a0-cf71638282b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 29 22:30:55.242002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.241967    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:55.500719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.500653    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"]
Oct 29 22:30:55.505923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.505891    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:55.507426 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.507401    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"]
Oct 29 22:30:55.618213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.618184    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b45a2214-j9pjp\" (UID: \"83847c71-2143-44d7-b57e-1dfab1cfad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:55.618433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.618417    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hln7\" (UniqueName: \"kubernetes.io/projected/83847c71-2143-44d7-b57e-1dfab1cfad9f-kube-api-access-8hln7\") pod \"maas-api-revoke-sub-b45a2214-j9pjp\" (UID: \"83847c71-2143-44d7-b57e-1dfab1cfad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:55.618692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.618681    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b45a2214-j9pjp\" (UID: \"83847c71-2143-44d7-b57e-1dfab1cfad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:55.704189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.704166    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:55.720192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.720126    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b45a2214-j9pjp\" (UID: \"83847c71-2143-44d7-b57e-1dfab1cfad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:55.720192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.720166    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hln7\" (UniqueName: \"kubernetes.io/projected/83847c71-2143-44d7-b57e-1dfab1cfad9f-kube-api-access-8hln7\") pod \"maas-api-revoke-sub-b45a2214-j9pjp\" (UID: \"83847c71-2143-44d7-b57e-1dfab1cfad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:55.720412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.720391    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b45a2214-j9pjp\" (UID: \"83847c71-2143-44d7-b57e-1dfab1cfad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:55.721004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.720982    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b45a2214-j9pjp\" (UID: \"83847c71-2143-44d7-b57e-1dfab1cfad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:55.722632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.722611    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b45a2214-j9pjp\" (UID: \"83847c71-2143-44d7-b57e-1dfab1cfad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:55.740708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.740684    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hln7\" (UniqueName: \"kubernetes.io/projected/83847c71-2143-44d7-b57e-1dfab1cfad9f-kube-api-access-8hln7\") pod \"maas-api-revoke-sub-b45a2214-j9pjp\" (UID: \"83847c71-2143-44d7-b57e-1dfab1cfad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:55.821371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.821309    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2018efa-a97e-4104-92ff-b97be01361e4-kube-api-access-lm8pf\" (UniqueName: \"kubernetes.io/projected/e2018efa-a97e-4104-92ff-b97be01361e4-kube-api-access-lm8pf\") pod \"e2018efa-a97e-4104-92ff-b97be01361e4\" (UID: \"e2018efa-a97e-4104-92ff-b97be01361e4\") "
Oct 29 22:30:55.821533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.821376    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-xks-ca\") pod \"e2018efa-a97e-4104-92ff-b97be01361e4\" (UID: \"e2018efa-a97e-4104-92ff-b97be01361e4\") "
Oct 29 22:30:55.821580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.821543    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-openshift-ca\") pod \"e2018efa-a97e-4104-92ff-b97be01361e4\" (UID: \"e2018efa-a97e-4104-92ff-b97be01361e4\") "
Oct 29 22:30:55.821887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.821861    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-openshift-ca" pod "e2018efa-a97e-4104-92ff-b97be01361e4" (UID: "e2018efa-a97e-4104-92ff-b97be01361e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:55.822030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.821994    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:55.823538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.823516    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-xks-ca" pod "e2018efa-a97e-4104-92ff-b97be01361e4" (UID: "e2018efa-a97e-4104-92ff-b97be01361e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:30:55.823538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.823524    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2018efa-a97e-4104-92ff-b97be01361e4-kube-api-access-lm8pf" pod "e2018efa-a97e-4104-92ff-b97be01361e4" (UID: "e2018efa-a97e-4104-92ff-b97be01361e4"). InnerVolumeSpecName "kube-api-access-lm8pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:55.826710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.826692    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:55.923069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.923038    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8pf\" (UniqueName: \"kubernetes.io/projected/e2018efa-a97e-4104-92ff-b97be01361e4-kube-api-access-lm8pf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:55.923069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:55.923070    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2018efa-a97e-4104-92ff-b97be01361e4-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:56.147827 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:30:56.147799    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83847c71_2143_44d7_b57e_1dfab1cfad9f.slice/crio-104c94ea2f6e32acfe6a3f45896d3a720af4dcaf4c5fa8926bf1c21fb2d54fec WatchSource:0}: Error finding container 104c94ea2f6e32acfe6a3f45896d3a720af4dcaf4c5fa8926bf1c21fb2d54fec: Status 404 returned error can't find the container with id 104c94ea2f6e32acfe6a3f45896d3a720af4dcaf4c5fa8926bf1c21fb2d54fec
Oct 29 22:30:56.148998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:56.148976    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"]
Oct 29 22:30:56.149812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:56.149793    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:30:56.149889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:56.149831    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" podUID="528f6325-67f0-490d-b4a0-cf71638282b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 29 22:30:56.492676 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:56.492633    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq" event={"ID":"e2018efa-a97e-4104-92ff-b97be01361e4","Type":"ContainerDied","Data":"be9e8e2b628d4f884c45dde287886c4630b1dd64c072ae302a5e6888f716b825"}
Oct 29 22:30:56.493144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:56.492703    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be9e8e2b628d4f884c45dde287886c4630b1dd64c072ae302a5e6888f716b825"
Oct 29 22:30:56.493144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:56.492767    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"
Oct 29 22:30:56.495966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:56.495930    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp" event={"ID":"83847c71-2143-44d7-b57e-1dfab1cfad9f","Type":"ContainerStarted","Data":"104c94ea2f6e32acfe6a3f45896d3a720af4dcaf4c5fa8926bf1c21fb2d54fec"}
Oct 29 22:30:56.942335 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:56.942284    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:57.150186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:57.150154    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:30:57.150374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:57.150206    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" podUID="528f6325-67f0-490d-b4a0-cf71638282b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 29 22:30:57.505769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:57.505688    4860 generic.go:356] "Generic (PLEG): container finished" podID="83847c71-2143-44d7-b57e-1dfab1cfad9f" containerID="debdd4c35749ba4d2b2ec7df27aea3db2b8fcb457476a1e36dc8d98eef6e377e" exitCode=0
Oct 29 22:30:57.506155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:57.505770    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp" event={"ID":"83847c71-2143-44d7-b57e-1dfab1cfad9f","Type":"ContainerDied","Data":"debdd4c35749ba4d2b2ec7df27aea3db2b8fcb457476a1e36dc8d98eef6e377e"}
Oct 29 22:30:58.150541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.150507    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 29 22:30:58.150711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.150568    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" podUID="528f6325-67f0-490d-b4a0-cf71638282b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 29 22:30:58.647949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.647909    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:30:58.701998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.701962    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:58.863622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.863526    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-xks-ca\") pod \"83847c71-2143-44d7-b57e-1dfab1cfad9f\" (UID: \"83847c71-2143-44d7-b57e-1dfab1cfad9f\") "
Oct 29 22:30:58.863778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.863645    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83847c71-2143-44d7-b57e-1dfab1cfad9f-kube-api-access-8hln7\" (UniqueName: \"kubernetes.io/projected/83847c71-2143-44d7-b57e-1dfab1cfad9f-kube-api-access-8hln7\") pod \"83847c71-2143-44d7-b57e-1dfab1cfad9f\" (UID: \"83847c71-2143-44d7-b57e-1dfab1cfad9f\") "
Oct 29 22:30:58.863778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.863695    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-openshift-ca\") pod \"83847c71-2143-44d7-b57e-1dfab1cfad9f\" (UID: \"83847c71-2143-44d7-b57e-1dfab1cfad9f\") "
Oct 29 22:30:58.864160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.864117    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-openshift-ca" pod "83847c71-2143-44d7-b57e-1dfab1cfad9f" (UID: "83847c71-2143-44d7-b57e-1dfab1cfad9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:30:58.865836 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.865808    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-xks-ca" pod "83847c71-2143-44d7-b57e-1dfab1cfad9f" (UID: "83847c71-2143-44d7-b57e-1dfab1cfad9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:30:58.865924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.865840    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83847c71-2143-44d7-b57e-1dfab1cfad9f-kube-api-access-8hln7" pod "83847c71-2143-44d7-b57e-1dfab1cfad9f" (UID: "83847c71-2143-44d7-b57e-1dfab1cfad9f"). InnerVolumeSpecName "kube-api-access-8hln7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:30:58.964750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.964713    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hln7\" (UniqueName: \"kubernetes.io/projected/83847c71-2143-44d7-b57e-1dfab1cfad9f-kube-api-access-8hln7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:58.964750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.964748    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:58.964750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:58.964759    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83847c71-2143-44d7-b57e-1dfab1cfad9f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:30:59.499920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:59.499818    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:59.500731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:59.500701    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:59.501654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:59.501609    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:30:59.522748 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:59.522730    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"
Oct 29 22:30:59.522859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:59.522789    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp" event={"ID":"83847c71-2143-44d7-b57e-1dfab1cfad9f","Type":"ContainerDied","Data":"104c94ea2f6e32acfe6a3f45896d3a720af4dcaf4c5fa8926bf1c21fb2d54fec"}
Oct 29 22:30:59.522859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:30:59.522842    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="104c94ea2f6e32acfe6a3f45896d3a720af4dcaf4c5fa8926bf1c21fb2d54fec"
Oct 29 22:31:00.369527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:00.369493    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:02.114281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.114246    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:02.964273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.964243    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"]
Oct 29 22:31:02.964835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.964821    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e2018efa-a97e-4104-92ff-b97be01361e4" containerName="revoke-keys"
Oct 29 22:31:02.964881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.964837    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2018efa-a97e-4104-92ff-b97be01361e4" containerName="revoke-keys"
Oct 29 22:31:02.964881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.964860    4860 cpu_manager.go:414] "containerMap: removing container" podUID="83847c71-2143-44d7-b57e-1dfab1cfad9f" containerName="revoke-keys"
Oct 29 22:31:02.964881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.964866    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83847c71-2143-44d7-b57e-1dfab1cfad9f" containerName="revoke-keys"
Oct 29 22:31:02.964973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.964950    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2018efa-a97e-4104-92ff-b97be01361e4" containerName="revoke-keys"
Oct 29 22:31:02.964973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.964962    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="83847c71-2143-44d7-b57e-1dfab1cfad9f" containerName="revoke-keys"
Oct 29 22:31:02.968916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.968901    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:02.986284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.986094    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"]
Oct 29 22:31:02.994732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.994710    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:31:02.994858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.994782    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:31:02.994858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.994845    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:31:02.998573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:02.998552    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:31:03.113069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:03.113033    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa7898f8-hgfmb\" (UID: \"8eb79b8a-b606-4347-9015-af441b1992a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:03.113264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:03.113189    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c25m\" (UniqueName: \"kubernetes.io/projected/8eb79b8a-b606-4347-9015-af441b1992a0-kube-api-access-8c25m\") pod \"maas-api-revoke-sub-fa7898f8-hgfmb\" (UID: \"8eb79b8a-b606-4347-9015-af441b1992a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:03.113264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:03.113244    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa7898f8-hgfmb\" (UID: \"8eb79b8a-b606-4347-9015-af441b1992a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:03.214298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:03.214225    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c25m\" (UniqueName: \"kubernetes.io/projected/8eb79b8a-b606-4347-9015-af441b1992a0-kube-api-access-8c25m\") pod \"maas-api-revoke-sub-fa7898f8-hgfmb\" (UID: \"8eb79b8a-b606-4347-9015-af441b1992a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:03.214298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:03.214283    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa7898f8-hgfmb\" (UID: \"8eb79b8a-b606-4347-9015-af441b1992a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:03.214793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:03.214388    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa7898f8-hgfmb\" (UID: \"8eb79b8a-b606-4347-9015-af441b1992a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:03.215127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:03.215102    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa7898f8-hgfmb\" (UID: \"8eb79b8a-b606-4347-9015-af441b1992a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:03.216815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:03.216793    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa7898f8-hgfmb\" (UID: \"8eb79b8a-b606-4347-9015-af441b1992a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:03.235664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:03.235643    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c25m\" (UniqueName: \"kubernetes.io/projected/8eb79b8a-b606-4347-9015-af441b1992a0-kube-api-access-8c25m\") pod \"maas-api-revoke-sub-fa7898f8-hgfmb\" (UID: \"8eb79b8a-b606-4347-9015-af441b1992a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:03.283084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:03.283064    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:03.584360 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:03.584304    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb79b8a_b606_4347_9015_af441b1992a0.slice/crio-0ecf400c7c32e2183cbf3496685ee42e16475127ae95438e08957333e1e35aec WatchSource:0}: Error finding container 0ecf400c7c32e2183cbf3496685ee42e16475127ae95438e08957333e1e35aec: Status 404 returned error can't find the container with id 0ecf400c7c32e2183cbf3496685ee42e16475127ae95438e08957333e1e35aec
Oct 29 22:31:03.587505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:03.587486    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"]
Oct 29 22:31:03.746577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:03.746547    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:04.561010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:04.560984    4860 generic.go:356] "Generic (PLEG): container finished" podID="8eb79b8a-b606-4347-9015-af441b1992a0" containerID="9d75ca73c938317ca706c9c3c05b31d7303f4d5bf5a631ae34d98d992c077769" exitCode=0
Oct 29 22:31:04.561349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:04.561026    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb" event={"ID":"8eb79b8a-b606-4347-9015-af441b1992a0","Type":"ContainerDied","Data":"9d75ca73c938317ca706c9c3c05b31d7303f4d5bf5a631ae34d98d992c077769"}
Oct 29 22:31:04.561349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:04.561082    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb" event={"ID":"8eb79b8a-b606-4347-9015-af441b1992a0","Type":"ContainerStarted","Data":"0ecf400c7c32e2183cbf3496685ee42e16475127ae95438e08957333e1e35aec"}
Oct 29 22:31:05.428788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.428755    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:05.479561 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.479523    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"]
Oct 29 22:31:05.484117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.484093    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.510683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.510654    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"]
Oct 29 22:31:05.620497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.620464    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"]
Oct 29 22:31:05.624942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.624917    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.645765 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.645737    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"]
Oct 29 22:31:05.647411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.646686    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.647411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.646774    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aabe95-a731-4377-966e-238d39d5ea99-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.647411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.646871    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.647411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.646907    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl2xd\" (UniqueName: \"kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-kube-api-access-wl2xd\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.647411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.646978    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.647411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.647049    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.647411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.647073    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.647411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.647154    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.647411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.647214    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aabe95-a731-4377-966e-238d39d5ea99-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.748801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.748767    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.748941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.748825    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl2xd\" (UniqueName: \"kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-kube-api-access-wl2xd\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.748941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.748872    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.748941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.748904    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.748941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.748929    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.749197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749034    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.749197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749103    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.749197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749129    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.749197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749176    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.749439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749201    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e41504b-834f-4cc2-89f9-ed6b7338f644-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.749439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749305    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.749439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749348    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.749439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749418    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aabe95-a731-4377-966e-238d39d5ea99-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.749635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749445    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.749635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749476    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.749635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749487    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.749635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749555    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.749635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749569    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.749871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749646    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.749871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749664    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aabe95-a731-4377-966e-238d39d5ea99-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.749871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749708    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjxt\" (UniqueName: \"kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-kube-api-access-vmjxt\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.750030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.749860    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.750337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.750308    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aabe95-a731-4377-966e-238d39d5ea99-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.751611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.751561    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.751997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.751975    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aabe95-a731-4377-966e-238d39d5ea99-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.755660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.755638    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:05.769425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.769399    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.769604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.769587    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl2xd\" (UniqueName: \"kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-kube-api-access-wl2xd\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.797857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.797835    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:05.850667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.850621    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.850805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.850692    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.850805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.850720    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjxt\" (UniqueName: \"kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-kube-api-access-vmjxt\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.850805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.850803    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.850933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.850826    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.850933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.850850    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.850933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.850897    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.850933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.850911    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e41504b-834f-4cc2-89f9-ed6b7338f644-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.851097 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.850955    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.851097 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.851072    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.851331 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.851296    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.852440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.851871    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.852440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.852154    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.854815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.852563    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e41504b-834f-4cc2-89f9-ed6b7338f644-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.854815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.853414    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.854815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.853731    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.873379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.872529    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.873379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.873147    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjxt\" (UniqueName: \"kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-kube-api-access-vmjxt\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:05.951485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.951458    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-openshift-ca\") pod \"8eb79b8a-b606-4347-9015-af441b1992a0\" (UID: \"8eb79b8a-b606-4347-9015-af441b1992a0\") "
Oct 29 22:31:05.951616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.951509    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-xks-ca\") pod \"8eb79b8a-b606-4347-9015-af441b1992a0\" (UID: \"8eb79b8a-b606-4347-9015-af441b1992a0\") "
Oct 29 22:31:05.951616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.951599    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb79b8a-b606-4347-9015-af441b1992a0-kube-api-access-8c25m\" (UniqueName: \"kubernetes.io/projected/8eb79b8a-b606-4347-9015-af441b1992a0-kube-api-access-8c25m\") pod \"8eb79b8a-b606-4347-9015-af441b1992a0\" (UID: \"8eb79b8a-b606-4347-9015-af441b1992a0\") "
Oct 29 22:31:05.951872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.951847    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-openshift-ca" pod "8eb79b8a-b606-4347-9015-af441b1992a0" (UID: "8eb79b8a-b606-4347-9015-af441b1992a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:31:05.952109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.952089    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:05.953501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.953477    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-xks-ca" pod "8eb79b8a-b606-4347-9015-af441b1992a0" (UID: "8eb79b8a-b606-4347-9015-af441b1992a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:31:05.953583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.953573    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb79b8a-b606-4347-9015-af441b1992a0-kube-api-access-8c25m" pod "8eb79b8a-b606-4347-9015-af441b1992a0" (UID: "8eb79b8a-b606-4347-9015-af441b1992a0"). InnerVolumeSpecName "kube-api-access-8c25m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:31:05.957744 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:05.957725    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:06.054194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.054159    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb79b8a-b606-4347-9015-af441b1992a0-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:06.054348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.054200    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c25m\" (UniqueName: \"kubernetes.io/projected/8eb79b8a-b606-4347-9015-af441b1992a0-kube-api-access-8c25m\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:06.112048 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:06.112017    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55aabe95_a731_4377_966e_238d39d5ea99.slice/crio-04f6092c3ba7a161c9d62ac5ec73749e2c182eea5b0490a7c381fce71f707aa1 WatchSource:0}: Error finding container 04f6092c3ba7a161c9d62ac5ec73749e2c182eea5b0490a7c381fce71f707aa1: Status 404 returned error can't find the container with id 04f6092c3ba7a161c9d62ac5ec73749e2c182eea5b0490a7c381fce71f707aa1
Oct 29 22:31:06.115032 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.114988    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:06.115132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.115083    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:06.115172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.115146    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:06.115379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.115359    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"]
Oct 29 22:31:06.259335 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:06.259291    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e41504b_834f_4cc2_89f9_ed6b7338f644.slice/crio-1888c197f2222c17ec528e1324609074d8d25e977de7db7886c354c25dfbe8ac WatchSource:0}: Error finding container 1888c197f2222c17ec528e1324609074d8d25e977de7db7886c354c25dfbe8ac: Status 404 returned error can't find the container with id 1888c197f2222c17ec528e1324609074d8d25e977de7db7886c354c25dfbe8ac
Oct 29 22:31:06.261921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.261853    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:06.262031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.261927    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:06.262031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.261954    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:06.263459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.263432    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"]
Oct 29 22:31:06.587737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.587294    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb" event={"ID":"8eb79b8a-b606-4347-9015-af441b1992a0","Type":"ContainerDied","Data":"0ecf400c7c32e2183cbf3496685ee42e16475127ae95438e08957333e1e35aec"}
Oct 29 22:31:06.587737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.587377    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"
Oct 29 22:31:06.587737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.587388    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ecf400c7c32e2183cbf3496685ee42e16475127ae95438e08957333e1e35aec"
Oct 29 22:31:06.589777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.589355    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" event={"ID":"4e41504b-834f-4cc2-89f9-ed6b7338f644","Type":"ContainerStarted","Data":"b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480"}
Oct 29 22:31:06.589777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.589407    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" event={"ID":"4e41504b-834f-4cc2-89f9-ed6b7338f644","Type":"ContainerStarted","Data":"1888c197f2222c17ec528e1324609074d8d25e977de7db7886c354c25dfbe8ac"}
Oct 29 22:31:06.591532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.591510    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" event={"ID":"55aabe95-a731-4377-966e-238d39d5ea99","Type":"ContainerStarted","Data":"ab83100751b821238e38a1d033dec3713018185db3798ad191d0906cad4680e1"}
Oct 29 22:31:06.591645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.591556    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" event={"ID":"55aabe95-a731-4377-966e-238d39d5ea99","Type":"ContainerStarted","Data":"04f6092c3ba7a161c9d62ac5ec73749e2c182eea5b0490a7c381fce71f707aa1"}
Oct 29 22:31:06.635151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.635098    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" podStartSLOduration=1.635081833 podStartE2EDuration="1.635081833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:06.632554049 +0000 UTC m=+1008.657330199" watchObservedRunningTime="2026-10-29 22:31:06.635081833 +0000 UTC m=+1008.659857969"
Oct 29 22:31:06.695583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.695534    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" podStartSLOduration=1.695516772 podStartE2EDuration="1.695516772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:06.693512947 +0000 UTC m=+1008.718289079" watchObservedRunningTime="2026-10-29 22:31:06.695516772 +0000 UTC m=+1008.720292905"
Oct 29 22:31:06.798987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.798956    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:06.958772 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:06.958740    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:07.254609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:07.254536    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:07.809348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:07.802702    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:31:07.809348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:07.802765    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 29 22:31:07.960452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:07.960418    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:31:07.960606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:07.960469    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:31:08.024779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.024750    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"]
Oct 29 22:31:08.025544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.025526    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8eb79b8a-b606-4347-9015-af441b1992a0" containerName="revoke-keys"
Oct 29 22:31:08.025660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.025649    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb79b8a-b606-4347-9015-af441b1992a0" containerName="revoke-keys"
Oct 29 22:31:08.025872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.025863    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb79b8a-b606-4347-9015-af441b1992a0" containerName="revoke-keys"
Oct 29 22:31:08.029769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.029752    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:08.030255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.029774    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"]
Oct 29 22:31:08.050438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.048685    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:31:08.050438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.048742    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:31:08.062506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.062449    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:31:08.062589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.062498    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:31:08.129186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.129145    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5ae3e8a-wp6mn\" (UID: \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:08.129399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.129346    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkpsz\" (UniqueName: \"kubernetes.io/projected/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-kube-api-access-qkpsz\") pod \"maas-api-revoke-sub-b5ae3e8a-wp6mn\" (UID: \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:08.129485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.129467    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5ae3e8a-wp6mn\" (UID: \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:08.230773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.230728    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkpsz\" (UniqueName: \"kubernetes.io/projected/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-kube-api-access-qkpsz\") pod \"maas-api-revoke-sub-b5ae3e8a-wp6mn\" (UID: \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:08.230939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.230879    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5ae3e8a-wp6mn\" (UID: \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:08.231017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.230998    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5ae3e8a-wp6mn\" (UID: \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:08.232381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.232264    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5ae3e8a-wp6mn\" (UID: \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:08.237048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.237017    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5ae3e8a-wp6mn\" (UID: \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:08.253861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.253835    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkpsz\" (UniqueName: \"kubernetes.io/projected/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-kube-api-access-qkpsz\") pod \"maas-api-revoke-sub-b5ae3e8a-wp6mn\" (UID: \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:08.351775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.351693    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:08.760111 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:08.760088    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fc3b71_f6ac_4940_bd0d_ef4b99e9eaf8.slice/crio-f084105ae8e38330e84b13862aa80f5864456775ed7e72f2a9b4264a079afbd3 WatchSource:0}: Error finding container f084105ae8e38330e84b13862aa80f5864456775ed7e72f2a9b4264a079afbd3: Status 404 returned error can't find the container with id f084105ae8e38330e84b13862aa80f5864456775ed7e72f2a9b4264a079afbd3
Oct 29 22:31:08.763423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.763397    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"]
Oct 29 22:31:08.808715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.808678    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:31:08.808951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.808924    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 29 22:31:08.958480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.958451    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:31:08.958922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:08.958893    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:31:09.112405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:09.112123    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:09.647780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:09.647745    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn" event={"ID":"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8","Type":"ContainerStarted","Data":"4cc47be3ab5839aac9601c5ee5e95acabb670c42c0bedddf8cefa0eef842689d"}
Oct 29 22:31:09.647956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:09.647797    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn" event={"ID":"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8","Type":"ContainerStarted","Data":"f084105ae8e38330e84b13862aa80f5864456775ed7e72f2a9b4264a079afbd3"}
Oct 29 22:31:09.720351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:09.719944    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn" podStartSLOduration=2.719926559 podStartE2EDuration="2.719926559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:09.690747438 +0000 UTC m=+1011.715523598" watchObservedRunningTime="2026-10-29 22:31:09.719926559 +0000 UTC m=+1011.744702690"
Oct 29 22:31:09.804350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:09.800667    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:31:09.804350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:09.800726    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 29 22:31:09.958416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:09.958385    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:31:09.958531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:09.958448    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:31:10.557690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:10.557658    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:10.657201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:10.657173    4860 generic.go:356] "Generic (PLEG): container finished" podID="68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8" containerID="4cc47be3ab5839aac9601c5ee5e95acabb670c42c0bedddf8cefa0eef842689d" exitCode=0
Oct 29 22:31:10.657342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:10.657256    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn" event={"ID":"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8","Type":"ContainerDied","Data":"4cc47be3ab5839aac9601c5ee5e95acabb670c42c0bedddf8cefa0eef842689d"}
Oct 29 22:31:10.818611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:10.818474    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:31:10.818873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:10.818831    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 29 22:31:10.959442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:10.959412    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:31:10.959896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:10.959862    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:31:10.977033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:10.976991    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:11.810690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:11.809717    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:31:11.810690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:11.809777    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 29 22:31:11.958030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:11.958001    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:31:11.958154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:11.958052    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:31:11.961655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:11.961634    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:12.027081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.027048    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-openshift-ca\") pod \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\" (UID: \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\") "
Oct 29 22:31:12.027355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.027222    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-xks-ca\") pod \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\" (UID: \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\") "
Oct 29 22:31:12.027430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.027365    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-kube-api-access-qkpsz\" (UniqueName: \"kubernetes.io/projected/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-kube-api-access-qkpsz\") pod \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\" (UID: \"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8\") "
Oct 29 22:31:12.027727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.027702    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-openshift-ca" pod "68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8" (UID: "68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:31:12.029307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.029279    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-xks-ca" pod "68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8" (UID: "68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:31:12.029657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.029627    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-kube-api-access-qkpsz" pod "68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8" (UID: "68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8"). InnerVolumeSpecName "kube-api-access-qkpsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:31:12.128633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.128599    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkpsz\" (UniqueName: \"kubernetes.io/projected/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-kube-api-access-qkpsz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:12.128633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.128632    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:12.128853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.128645    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:12.690662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.690624    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn" event={"ID":"68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8","Type":"ContainerDied","Data":"f084105ae8e38330e84b13862aa80f5864456775ed7e72f2a9b4264a079afbd3"}
Oct 29 22:31:12.690954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.690922    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f084105ae8e38330e84b13862aa80f5864456775ed7e72f2a9b4264a079afbd3"
Oct 29 22:31:12.691151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.691136    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"
Oct 29 22:31:12.771856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.771822    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"]
Oct 29 22:31:12.772406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.772081    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57" podUID="24145e7e-9d32-42a0-a462-dca182830c13" containerName="istio-proxy" containerID="cri-o://5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e" gracePeriod=30
Oct 29 22:31:12.811536 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.800980    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:31:12.811536 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.801035    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 29 22:31:12.959147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.959051    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:31:12.959147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.959116    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:31:12.971788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:12.971760    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:13.271395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.271294    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"]
Oct 29 22:31:13.272746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.272722    4860 cpu_manager.go:414] "containerMap: removing container" podUID="68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8" containerName="revoke-keys"
Oct 29 22:31:13.272898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.272884    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8" containerName="revoke-keys"
Oct 29 22:31:13.273187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.273174    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8" containerName="revoke-keys"
Oct 29 22:31:13.296190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.296162    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:13.317389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.317356    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"]
Oct 29 22:31:13.353596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.352808    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:31:13.353596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.352970    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:31:13.353596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.353048    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:31:13.353596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.353129    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:31:13.495911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.495870    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdmxv\" (UniqueName: \"kubernetes.io/projected/936d0568-d56a-4fd0-99c5-51f769ffd518-kube-api-access-cdmxv\") pod \"maas-api-revoke-sub-834c33af-65b7f\" (UID: \"936d0568-d56a-4fd0-99c5-51f769ffd518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:13.496291 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.496268    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-834c33af-65b7f\" (UID: \"936d0568-d56a-4fd0-99c5-51f769ffd518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:13.496479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.496461    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-834c33af-65b7f\" (UID: \"936d0568-d56a-4fd0-99c5-51f769ffd518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:13.602458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.602352    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdmxv\" (UniqueName: \"kubernetes.io/projected/936d0568-d56a-4fd0-99c5-51f769ffd518-kube-api-access-cdmxv\") pod \"maas-api-revoke-sub-834c33af-65b7f\" (UID: \"936d0568-d56a-4fd0-99c5-51f769ffd518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:13.603542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.603513    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-834c33af-65b7f\" (UID: \"936d0568-d56a-4fd0-99c5-51f769ffd518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:13.604208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.604188    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-834c33af-65b7f\" (UID: \"936d0568-d56a-4fd0-99c5-51f769ffd518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:13.605313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.605108    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-834c33af-65b7f\" (UID: \"936d0568-d56a-4fd0-99c5-51f769ffd518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:13.804349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.803433    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:31:13.804349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.803510    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 29 22:31:13.804599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.804436    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-834c33af-65b7f\" (UID: \"936d0568-d56a-4fd0-99c5-51f769ffd518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:13.896396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.887638    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdmxv\" (UniqueName: \"kubernetes.io/projected/936d0568-d56a-4fd0-99c5-51f769ffd518-kube-api-access-cdmxv\") pod \"maas-api-revoke-sub-834c33af-65b7f\" (UID: \"936d0568-d56a-4fd0-99c5-51f769ffd518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:13.963353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.961711    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:31:13.963353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:13.961771    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:31:14.048349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:14.043188    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:14.453349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:14.446449    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:14.475217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:14.475116    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Oct 29 22:31:14.544349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:14.541810    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.068507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.068459    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:31:15.069632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.068407    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.175184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.174730    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.217845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.217805    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.245170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.244138    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.268749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.268721    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:15.268965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.268941    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.317988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.316439    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.446728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.446688    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.448006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.447986    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:31:15.489752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.488764    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"]
Oct 29 22:31:15.490660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.490629    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Oct 29 22:31:15.492670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.492651    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:31:15.558340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.558307    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:15.562385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.557823    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:31:15.562786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.552807    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:15.562991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.562937    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" podUID="528f6325-67f0-490d-b4a0-cf71638282b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.585482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.563899    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:31:15.586689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.585785    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:15.586857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.564992    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.595607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.595590    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:31:15.768829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.768491    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:15.778475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.774459    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" podUID="60189037-441a-4cb7-866c-6287bda16518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:15.905502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:15.905470    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:31:16.008934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.008893    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:31:16.024810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.024740    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:16.025004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.024975    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.076796 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.076762    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:31:16.078339 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.078302    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:31:16.112466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.111923    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.135421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.135381    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.181018 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.180987    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:16.198571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.198531    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.218653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.218604    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:31:16.295136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.295011    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:31:16.329157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.328435    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.329837 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.329551    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.424841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.424806    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f" event={"ID":"936d0568-d56a-4fd0-99c5-51f769ffd518","Type":"ContainerStarted","Data":"910f9b5d2eb00694992fef6a144bf143e456e15f0e4d6150f5e3c900ca18ffbc"}
Oct 29 22:31:16.425210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.425192    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:31:16.451275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.451252    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:31:16.452914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.452320    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:31:16.453137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.453122    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:16.453892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.453868    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:31:16.461880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.461864    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:31:16.462049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.462036    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:31:16.462752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.462738    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:31:16.463446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.463432    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:16.469530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.469512    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:31:16.470052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.470036    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:31:16.583086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.582989    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.615728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.615689    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:31:16.617146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.617121    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:31:16.631873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.630586    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:16.652133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.652106    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:31:16.701112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.701081    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:31:16.703544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.703519    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:31:16.743530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:16.743484    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:31:17.201365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:17.198651    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f" event={"ID":"936d0568-d56a-4fd0-99c5-51f769ffd518","Type":"ContainerStarted","Data":"06d0fb205450f0f15bc7f1f4a2a272b94c46f614359431d8aa71f41123c316d2"}
Oct 29 22:31:17.976041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:17.975887    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:18.033002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.032967    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:31:18.050810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.050741    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-credential-socket\") pod \"24145e7e-9d32-42a0-a462-dca182830c13\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") "
Oct 29 22:31:18.051029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.051009    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-istio-token\" (UniqueName: \"kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-istio-token\") pod \"24145e7e-9d32-42a0-a462-dca182830c13\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") "
Oct 29 22:31:18.051305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.051284    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-data\") pod \"24145e7e-9d32-42a0-a462-dca182830c13\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") "
Oct 29 22:31:18.051587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.051567    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-socket\") pod \"24145e7e-9d32-42a0-a462-dca182830c13\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") "
Oct 29 22:31:18.051752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.051050    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-credential-socket" pod "24145e7e-9d32-42a0-a462-dca182830c13" (UID: "24145e7e-9d32-42a0-a462-dca182830c13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:31:18.051816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.051612    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-data" pod "24145e7e-9d32-42a0-a462-dca182830c13" (UID: "24145e7e-9d32-42a0-a462-dca182830c13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:31:18.051816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.051755    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-socket" pod "24145e7e-9d32-42a0-a462-dca182830c13" (UID: "24145e7e-9d32-42a0-a462-dca182830c13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:31:18.051930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.051909    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-envoy\") pod \"24145e7e-9d32-42a0-a462-dca182830c13\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") "
Oct 29 22:31:18.052050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.051951    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24145e7e-9d32-42a0-a462-dca182830c13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24145e7e-9d32-42a0-a462-dca182830c13-istiod-ca-cert\") pod \"24145e7e-9d32-42a0-a462-dca182830c13\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") "
Oct 29 22:31:18.052050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.051984    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-kube-api-access-rll8j\" (UniqueName: \"kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-kube-api-access-rll8j\") pod \"24145e7e-9d32-42a0-a462-dca182830c13\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") "
Oct 29 22:31:18.052601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.052575    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24145e7e-9d32-42a0-a462-dca182830c13-istiod-ca-cert" pod "24145e7e-9d32-42a0-a462-dca182830c13" (UID: "24145e7e-9d32-42a0-a462-dca182830c13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:31:18.052688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.052603    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:18.052688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.052621    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:18.052688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.052631    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:18.054186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.054165    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-envoy" pod "24145e7e-9d32-42a0-a462-dca182830c13" (UID: "24145e7e-9d32-42a0-a462-dca182830c13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:31:18.054477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.054457    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-istio-token" pod "24145e7e-9d32-42a0-a462-dca182830c13" (UID: "24145e7e-9d32-42a0-a462-dca182830c13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:31:18.055291 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.055224    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-kube-api-access-rll8j" pod "24145e7e-9d32-42a0-a462-dca182830c13" (UID: "24145e7e-9d32-42a0-a462-dca182830c13"). InnerVolumeSpecName "kube-api-access-rll8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:31:18.059549 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:31:18.059521    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:31:18.084550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.084507    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f" podStartSLOduration=5.084490943 podStartE2EDuration="5.084490943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:17.244557748 +0000 UTC m=+1019.269333907" watchObservedRunningTime="2026-10-29 22:31:18.084490943 +0000 UTC m=+1020.109267076"
Oct 29 22:31:18.153408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.153377    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24145e7e-9d32-42a0-a462-dca182830c13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24145e7e-9d32-42a0-a462-dca182830c13-istio-podinfo\") pod \"24145e7e-9d32-42a0-a462-dca182830c13\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") "
Oct 29 22:31:18.153591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.153419    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-certs\") pod \"24145e7e-9d32-42a0-a462-dca182830c13\" (UID: \"24145e7e-9d32-42a0-a462-dca182830c13\") "
Oct 29 22:31:18.153750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.153722    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-certs" pod "24145e7e-9d32-42a0-a462-dca182830c13" (UID: "24145e7e-9d32-42a0-a462-dca182830c13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:31:18.153851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.153819    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:18.153851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.153831    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:18.153851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.153839    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24145e7e-9d32-42a0-a462-dca182830c13-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:18.153851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.153846    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24145e7e-9d32-42a0-a462-dca182830c13-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:18.153978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.153860    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rll8j\" (UniqueName: \"kubernetes.io/projected/24145e7e-9d32-42a0-a462-dca182830c13-kube-api-access-rll8j\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:18.155400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.155375    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24145e7e-9d32-42a0-a462-dca182830c13-istio-podinfo" pod "24145e7e-9d32-42a0-a462-dca182830c13" (UID: "24145e7e-9d32-42a0-a462-dca182830c13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:31:18.209594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.209563    4860 generic.go:356] "Generic (PLEG): container finished" podID="24145e7e-9d32-42a0-a462-dca182830c13" containerID="5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e" exitCode=0
Oct 29 22:31:18.210100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.209851    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"
Oct 29 22:31:18.210427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.210397    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57" event={"ID":"24145e7e-9d32-42a0-a462-dca182830c13","Type":"ContainerDied","Data":"5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e"}
Oct 29 22:31:18.210548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.210467    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57" event={"ID":"24145e7e-9d32-42a0-a462-dca182830c13","Type":"ContainerDied","Data":"0bf2dcf97e7ec66a5375c63f9d3a18179a32c260e4fe70d0e5188199d98f0bf6"}
Oct 29 22:31:18.210548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.210499    4860 scope.go:122] "RemoveContainer" containerID="5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e"
Oct 29 22:31:18.224192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.222668    4860 generic.go:356] "Generic (PLEG): container finished" podID="936d0568-d56a-4fd0-99c5-51f769ffd518" containerID="06d0fb205450f0f15bc7f1f4a2a272b94c46f614359431d8aa71f41123c316d2" exitCode=0
Oct 29 22:31:18.224192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.223885    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f" event={"ID":"936d0568-d56a-4fd0-99c5-51f769ffd518","Type":"ContainerDied","Data":"06d0fb205450f0f15bc7f1f4a2a272b94c46f614359431d8aa71f41123c316d2"}
Oct 29 22:31:18.229577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.229556    4860 scope.go:122] "RemoveContainer" containerID="5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e"
Oct 29 22:31:18.229970 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:31:18.229947    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e\": container with ID starting with 5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e not found: ID does not exist" containerID="5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e"
Oct 29 22:31:18.230045 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.229997    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e"} err="failed to get container status \"5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e\": rpc error: code = NotFound desc = could not find container \"5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e\": container with ID starting with 5c057185c94990870ca826ce2229aadc699bf40e20da9e14671040291b9c606e not found: ID does not exist"
Oct 29 22:31:18.255925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.255890    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24145e7e-9d32-42a0-a462-dca182830c13-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:18.271267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.271244    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"]
Oct 29 22:31:18.279841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:18.279816    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-fhw57"]
Oct 29 22:31:19.385926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:19.385894    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:19.471142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:19.471105    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-xks-ca\") pod \"936d0568-d56a-4fd0-99c5-51f769ffd518\" (UID: \"936d0568-d56a-4fd0-99c5-51f769ffd518\") "
Oct 29 22:31:19.471302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:19.471221    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-openshift-ca\") pod \"936d0568-d56a-4fd0-99c5-51f769ffd518\" (UID: \"936d0568-d56a-4fd0-99c5-51f769ffd518\") "
Oct 29 22:31:19.471302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:19.471252    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936d0568-d56a-4fd0-99c5-51f769ffd518-kube-api-access-cdmxv\" (UniqueName: \"kubernetes.io/projected/936d0568-d56a-4fd0-99c5-51f769ffd518-kube-api-access-cdmxv\") pod \"936d0568-d56a-4fd0-99c5-51f769ffd518\" (UID: \"936d0568-d56a-4fd0-99c5-51f769ffd518\") "
Oct 29 22:31:19.471624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:19.471601    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-openshift-ca" pod "936d0568-d56a-4fd0-99c5-51f769ffd518" (UID: "936d0568-d56a-4fd0-99c5-51f769ffd518"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:31:19.471714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:19.471700    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:19.473403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:19.473379    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-xks-ca" pod "936d0568-d56a-4fd0-99c5-51f769ffd518" (UID: "936d0568-d56a-4fd0-99c5-51f769ffd518"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:31:19.473597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:19.473572    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936d0568-d56a-4fd0-99c5-51f769ffd518-kube-api-access-cdmxv" pod "936d0568-d56a-4fd0-99c5-51f769ffd518" (UID: "936d0568-d56a-4fd0-99c5-51f769ffd518"). InnerVolumeSpecName "kube-api-access-cdmxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:31:19.572557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:19.572530    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdmxv\" (UniqueName: \"kubernetes.io/projected/936d0568-d56a-4fd0-99c5-51f769ffd518-kube-api-access-cdmxv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:19.572557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:19.572553    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936d0568-d56a-4fd0-99c5-51f769ffd518-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:19.802498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:19.802417    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:20.113275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:20.113194    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24145e7e-9d32-42a0-a462-dca182830c13" path="/var/lib/kubelet/pods/24145e7e-9d32-42a0-a462-dca182830c13/volumes"
Oct 29 22:31:20.240482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:20.240453    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"
Oct 29 22:31:20.240642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:20.240449    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f" event={"ID":"936d0568-d56a-4fd0-99c5-51f769ffd518","Type":"ContainerDied","Data":"910f9b5d2eb00694992fef6a144bf143e456e15f0e4d6150f5e3c900ca18ffbc"}
Oct 29 22:31:20.240642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:20.240579    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="910f9b5d2eb00694992fef6a144bf143e456e15f0e4d6150f5e3c900ca18ffbc"
Oct 29 22:31:21.073414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.073377    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"]
Oct 29 22:31:21.074179 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.074154    4860 cpu_manager.go:414] "containerMap: removing container" podUID="24145e7e-9d32-42a0-a462-dca182830c13" containerName="istio-proxy"
Oct 29 22:31:21.074179 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.074176    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24145e7e-9d32-42a0-a462-dca182830c13" containerName="istio-proxy"
Oct 29 22:31:21.074415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.074195    4860 cpu_manager.go:414] "containerMap: removing container" podUID="936d0568-d56a-4fd0-99c5-51f769ffd518" containerName="revoke-keys"
Oct 29 22:31:21.074415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.074203    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936d0568-d56a-4fd0-99c5-51f769ffd518" containerName="revoke-keys"
Oct 29 22:31:21.074415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.074398    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="24145e7e-9d32-42a0-a462-dca182830c13" containerName="istio-proxy"
Oct 29 22:31:21.074415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.074413    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="936d0568-d56a-4fd0-99c5-51f769ffd518" containerName="revoke-keys"
Oct 29 22:31:21.078426 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.078407    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.115349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.113483    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"]
Oct 29 22:31:21.190961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.190926    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.191102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.190970    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.191102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.190998    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.191193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.191165    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kn6r\" (UniqueName: \"kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-kube-api-access-7kn6r\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.191237 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.191223    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.191269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.191255    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.191300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.191281    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.191378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.191357    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.191507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.191495    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.292918 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.292888    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kn6r\" (UniqueName: \"kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-kube-api-access-7kn6r\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.292941    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.292975    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.293008    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.293031    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.293118    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.293182    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.293214    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.293346    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.293439    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.293595    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.293680    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.293784    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.293914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.293871    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.295743 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.295718    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.295920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.295902    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.314892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.314871    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.314981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.314874    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kn6r\" (UniqueName: \"kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-kube-api-access-7kn6r\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.393608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.393544    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:21.395213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.395190    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"]
Oct 29 22:31:21.399299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.399284    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:21.415976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.415954    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:31:21.416091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.416003    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:31:21.416091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.416028    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:31:21.416091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.416035    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:31:21.418889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.418689    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"]
Oct 29 22:31:21.495134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.495107    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e7dbcce-j65pg\" (UID: \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:21.495290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.495270    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmtsz\" (UniqueName: \"kubernetes.io/projected/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-kube-api-access-lmtsz\") pod \"maas-api-revoke-sub-2e7dbcce-j65pg\" (UID: \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:21.495425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.495409    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e7dbcce-j65pg\" (UID: \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:21.525639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.525607    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:21.597131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.597081    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e7dbcce-j65pg\" (UID: \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:21.597414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.597269    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmtsz\" (UniqueName: \"kubernetes.io/projected/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-kube-api-access-lmtsz\") pod \"maas-api-revoke-sub-2e7dbcce-j65pg\" (UID: \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:21.597513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.597421    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e7dbcce-j65pg\" (UID: \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:21.597976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.597926    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e7dbcce-j65pg\" (UID: \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:21.600158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.600137    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e7dbcce-j65pg\" (UID: \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:21.618759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.618732    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmtsz\" (UniqueName: \"kubernetes.io/projected/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-kube-api-access-lmtsz\") pod \"maas-api-revoke-sub-2e7dbcce-j65pg\" (UID: \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:21.717755 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:21.717728    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722561f0_eacd_40b5_8d78_8a67b99bdf7f.slice/crio-42e711e0d8874d2cbb94368329f0d2c5033fd32c6746b4490d195b8a52c05b05 WatchSource:0}: Error finding container 42e711e0d8874d2cbb94368329f0d2c5033fd32c6746b4490d195b8a52c05b05: Status 404 returned error can't find the container with id 42e711e0d8874d2cbb94368329f0d2c5033fd32c6746b4490d195b8a52c05b05
Oct 29 22:31:21.720714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.720678    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:21.720804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.720761    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:21.720873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.720805    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:21.722611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.722587    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"]
Oct 29 22:31:21.746694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:21.746676    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:22.057192 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:22.057160    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0935a1_4bdb_4990_bd2c_3145cff30d9c.slice/crio-6225a19b767aa786d1f93812e146994c03fc918e812c1c96765986a95b935aa3 WatchSource:0}: Error finding container 6225a19b767aa786d1f93812e146994c03fc918e812c1c96765986a95b935aa3: Status 404 returned error can't find the container with id 6225a19b767aa786d1f93812e146994c03fc918e812c1c96765986a95b935aa3
Oct 29 22:31:22.060735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.060711    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"]
Oct 29 22:31:22.258769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.258735    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg" event={"ID":"2a0935a1-4bdb-4990-bd2c-3145cff30d9c","Type":"ContainerStarted","Data":"6225a19b767aa786d1f93812e146994c03fc918e812c1c96765986a95b935aa3"}
Oct 29 22:31:22.260585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.260556    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" event={"ID":"722561f0-eacd-40b5-8d78-8a67b99bdf7f","Type":"ContainerStarted","Data":"1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5"}
Oct 29 22:31:22.260703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.260601    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" event={"ID":"722561f0-eacd-40b5-8d78-8a67b99bdf7f","Type":"ContainerStarted","Data":"42e711e0d8874d2cbb94368329f0d2c5033fd32c6746b4490d195b8a52c05b05"}
Oct 29 22:31:22.309289 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.309216    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" podStartSLOduration=1.309202588 podStartE2EDuration="1.309202588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:22.304890918 +0000 UTC m=+1024.329667090" watchObservedRunningTime="2026-10-29 22:31:22.309202588 +0000 UTC m=+1024.333978720"
Oct 29 22:31:22.394730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.394709    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:22.395862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.395840    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:31:22.395965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.395887    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 29 22:31:22.633215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.633138    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"]
Oct 29 22:31:22.637531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.637513    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:22.656316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.656291    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"]
Oct 29 22:31:22.713046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.713015    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c1234ae-lvjls\" (UID: \"cc8243fc-764b-46d3-98f4-65944b4bb80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:22.713247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.713085    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c1234ae-lvjls\" (UID: \"cc8243fc-764b-46d3-98f4-65944b4bb80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:22.713247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.713134    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzpxd\" (UniqueName: \"kubernetes.io/projected/cc8243fc-764b-46d3-98f4-65944b4bb80b-kube-api-access-qzpxd\") pod \"maas-api-revoke-sub-1c1234ae-lvjls\" (UID: \"cc8243fc-764b-46d3-98f4-65944b4bb80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:22.814371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.814318    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c1234ae-lvjls\" (UID: \"cc8243fc-764b-46d3-98f4-65944b4bb80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:22.814539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.814436    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzpxd\" (UniqueName: \"kubernetes.io/projected/cc8243fc-764b-46d3-98f4-65944b4bb80b-kube-api-access-qzpxd\") pod \"maas-api-revoke-sub-1c1234ae-lvjls\" (UID: \"cc8243fc-764b-46d3-98f4-65944b4bb80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:22.814642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.814626    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c1234ae-lvjls\" (UID: \"cc8243fc-764b-46d3-98f4-65944b4bb80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:22.815025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.815004    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c1234ae-lvjls\" (UID: \"cc8243fc-764b-46d3-98f4-65944b4bb80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:22.816971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.816950    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c1234ae-lvjls\" (UID: \"cc8243fc-764b-46d3-98f4-65944b4bb80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:22.835649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.835622    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzpxd\" (UniqueName: \"kubernetes.io/projected/cc8243fc-764b-46d3-98f4-65944b4bb80b-kube-api-access-qzpxd\") pod \"maas-api-revoke-sub-1c1234ae-lvjls\" (UID: \"cc8243fc-764b-46d3-98f4-65944b4bb80b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:22.952751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:22.952723    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:23.296987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:23.296954    4860 generic.go:356] "Generic (PLEG): container finished" podID="2a0935a1-4bdb-4990-bd2c-3145cff30d9c" containerID="454c1029a72150d44be1f6142b8997caa5bd4df7c7dad3208c23fdb2adc9f1c2" exitCode=0
Oct 29 22:31:23.297418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:23.297314    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg" event={"ID":"2a0935a1-4bdb-4990-bd2c-3145cff30d9c","Type":"ContainerDied","Data":"454c1029a72150d44be1f6142b8997caa5bd4df7c7dad3208c23fdb2adc9f1c2"}
Oct 29 22:31:23.297812 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:23.297785    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8243fc_764b_46d3_98f4_65944b4bb80b.slice/crio-8e0dece0a2d499aa97854ba4459192b0c5ee775851df2c8419648aa7ff737167 WatchSource:0}: Error finding container 8e0dece0a2d499aa97854ba4459192b0c5ee775851df2c8419648aa7ff737167: Status 404 returned error can't find the container with id 8e0dece0a2d499aa97854ba4459192b0c5ee775851df2c8419648aa7ff737167
Oct 29 22:31:23.300171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:23.300149    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"]
Oct 29 22:31:23.394993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:23.394941    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:31:23.395192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:23.395045    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 29 22:31:23.397181 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:23.397148    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:24.306577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.306548    4860 generic.go:356] "Generic (PLEG): container finished" podID="cc8243fc-764b-46d3-98f4-65944b4bb80b" containerID="4907a7d2a599eedf383905580508f4bdc26cfcc925fc813d99cb9e145147a453" exitCode=0
Oct 29 22:31:24.306901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.306601    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls" event={"ID":"cc8243fc-764b-46d3-98f4-65944b4bb80b","Type":"ContainerDied","Data":"4907a7d2a599eedf383905580508f4bdc26cfcc925fc813d99cb9e145147a453"}
Oct 29 22:31:24.306901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.306657    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls" event={"ID":"cc8243fc-764b-46d3-98f4-65944b4bb80b","Type":"ContainerStarted","Data":"8e0dece0a2d499aa97854ba4459192b0c5ee775851df2c8419648aa7ff737167"}
Oct 29 22:31:24.400341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.399877    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:31:24.400341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.399932    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 29 22:31:24.516683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.516652    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:24.638741 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.638662    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-openshift-ca\") pod \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\" (UID: \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\") "
Oct 29 22:31:24.638894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.638815    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-xks-ca\") pod \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\" (UID: \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\") "
Oct 29 22:31:24.638894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.638881    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-kube-api-access-lmtsz\" (UniqueName: \"kubernetes.io/projected/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-kube-api-access-lmtsz\") pod \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\" (UID: \"2a0935a1-4bdb-4990-bd2c-3145cff30d9c\") "
Oct 29 22:31:24.639033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.639004    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-openshift-ca" pod "2a0935a1-4bdb-4990-bd2c-3145cff30d9c" (UID: "2a0935a1-4bdb-4990-bd2c-3145cff30d9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:31:24.639432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.639414    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:24.640910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.640888    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-xks-ca" pod "2a0935a1-4bdb-4990-bd2c-3145cff30d9c" (UID: "2a0935a1-4bdb-4990-bd2c-3145cff30d9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:31:24.641050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.641031    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-kube-api-access-lmtsz" pod "2a0935a1-4bdb-4990-bd2c-3145cff30d9c" (UID: "2a0935a1-4bdb-4990-bd2c-3145cff30d9c"). InnerVolumeSpecName "kube-api-access-lmtsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:31:24.740317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.740289    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:24.740317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:24.740316    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmtsz\" (UniqueName: \"kubernetes.io/projected/2a0935a1-4bdb-4990-bd2c-3145cff30d9c-kube-api-access-lmtsz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:25.209461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.209428    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:25.317039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.317007    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"
Oct 29 22:31:25.317565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.317039    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg" event={"ID":"2a0935a1-4bdb-4990-bd2c-3145cff30d9c","Type":"ContainerDied","Data":"6225a19b767aa786d1f93812e146994c03fc918e812c1c96765986a95b935aa3"}
Oct 29 22:31:25.317565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.317088    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6225a19b767aa786d1f93812e146994c03fc918e812c1c96765986a95b935aa3"
Oct 29 22:31:25.395862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.395129    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:31:25.395862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.395187    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 29 22:31:25.532429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.532394    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:25.652716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.652688    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8243fc-764b-46d3-98f4-65944b4bb80b-kube-api-access-qzpxd\" (UniqueName: \"kubernetes.io/projected/cc8243fc-764b-46d3-98f4-65944b4bb80b-kube-api-access-qzpxd\") pod \"cc8243fc-764b-46d3-98f4-65944b4bb80b\" (UID: \"cc8243fc-764b-46d3-98f4-65944b4bb80b\") "
Oct 29 22:31:25.652716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.652720    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-xks-ca\") pod \"cc8243fc-764b-46d3-98f4-65944b4bb80b\" (UID: \"cc8243fc-764b-46d3-98f4-65944b4bb80b\") "
Oct 29 22:31:25.652926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.652843    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-openshift-ca\") pod \"cc8243fc-764b-46d3-98f4-65944b4bb80b\" (UID: \"cc8243fc-764b-46d3-98f4-65944b4bb80b\") "
Oct 29 22:31:25.653226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.653191    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-openshift-ca" pod "cc8243fc-764b-46d3-98f4-65944b4bb80b" (UID: "cc8243fc-764b-46d3-98f4-65944b4bb80b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:31:25.653493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.653444    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:25.654902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.654880    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8243fc-764b-46d3-98f4-65944b4bb80b-kube-api-access-qzpxd" pod "cc8243fc-764b-46d3-98f4-65944b4bb80b" (UID: "cc8243fc-764b-46d3-98f4-65944b4bb80b"). InnerVolumeSpecName "kube-api-access-qzpxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:31:25.655008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.654966    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-xks-ca" pod "cc8243fc-764b-46d3-98f4-65944b4bb80b" (UID: "cc8243fc-764b-46d3-98f4-65944b4bb80b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:31:25.755303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.755233    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzpxd\" (UniqueName: \"kubernetes.io/projected/cc8243fc-764b-46d3-98f4-65944b4bb80b-kube-api-access-qzpxd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:25.755303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:25.755260    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8243fc-764b-46d3-98f4-65944b4bb80b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:26.326355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:26.326290    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls" event={"ID":"cc8243fc-764b-46d3-98f4-65944b4bb80b","Type":"ContainerDied","Data":"8e0dece0a2d499aa97854ba4459192b0c5ee775851df2c8419648aa7ff737167"}
Oct 29 22:31:26.326747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:26.326377    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e0dece0a2d499aa97854ba4459192b0c5ee775851df2c8419648aa7ff737167"
Oct 29 22:31:26.326747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:26.326311    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"
Oct 29 22:31:26.410403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:26.409749    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:31:26.410403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:26.409811    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 29 22:31:27.400733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:27.400695    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:31:27.401312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:27.401278    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 29 22:31:27.677389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:27.677286    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:27.906521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:27.906464    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:27.985423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:27.984428    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.115495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.107271    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:31:28.147722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.147400    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:31:28.454346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.439855    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:31:28.454346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.439922    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 29 22:31:28.454346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.440390    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.477178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.477133    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Oct 29 22:31:28.481685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.478396    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.621488 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.621422    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.622908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.622865    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.648761 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.648409    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:28.648761 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.648469    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" podUID="fb326055-bb08-42f0-8597-ddfe942b2ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.661509 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.661475    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.723403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.712663    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.748347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.747509    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:28.914541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.914498    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:31:28.919346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.915355    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:31:29.046397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:28.991799    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:31:29.046397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.037625    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:31:29.221490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.221437    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:29.261348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.233508    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:29.261348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.233858    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:31:29.261348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.233920    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:31:29.417351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.413438    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Oct 29 22:31:29.417614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.417582    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Oct 29 22:31:29.480408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.465680    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:31:29.491283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.491251    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:31:29.524116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.479308    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:29.524116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.497986    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:31:29.560457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.560414    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:29.628099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.627918    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:31:29.629177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.628012    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:29.630375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.630355    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:31:29.630537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.630524    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:31:29.663489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.663418    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:29.664696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.664677    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:31:29.726391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.712903    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:29.727862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.727840    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:31:29.787345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.757775    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:31:29.791363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.790383    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:29.809353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.791686    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:31:29.809353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.802132    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:31:29.813947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.813929    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:31:29.923090 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.923013    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:31:29.973900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:29.973872    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:31:30.085306 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:30.085267    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:31:30.090305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:30.090283    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:31:30.090622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:30.090605    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:31:30.090888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:30.090875    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:31:30.091099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:30.088712    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": context deadline exceeded"
Oct 29 22:31:30.096468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:30.096026    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:31:30.152194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:30.152164    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:31:30.363271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:30.362877    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:30.763961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:30.763928    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:30.765304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:30.765279    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:30.766190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:30.766171    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:31:32.035110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.035077    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"]
Oct 29 22:31:32.035691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.035677    4860 cpu_manager.go:414] "containerMap: removing container" podUID="cc8243fc-764b-46d3-98f4-65944b4bb80b" containerName="revoke-keys"
Oct 29 22:31:32.035691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.035692    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8243fc-764b-46d3-98f4-65944b4bb80b" containerName="revoke-keys"
Oct 29 22:31:32.035767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.035706    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2a0935a1-4bdb-4990-bd2c-3145cff30d9c" containerName="revoke-keys"
Oct 29 22:31:32.035767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.035711    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0935a1-4bdb-4990-bd2c-3145cff30d9c" containerName="revoke-keys"
Oct 29 22:31:32.035832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.035811    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8243fc-764b-46d3-98f4-65944b4bb80b" containerName="revoke-keys"
Oct 29 22:31:32.035832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.035820    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0935a1-4bdb-4990-bd2c-3145cff30d9c" containerName="revoke-keys"
Oct 29 22:31:32.040852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.040835    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:32.059371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.059346    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"]
Oct 29 22:31:32.066456 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.066436    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:31:32.066576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.066512    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:31:32.066576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.066532    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:31:32.066576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.066538    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:31:32.107069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.107036    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfnvz\" (UniqueName: \"kubernetes.io/projected/dd2068b1-9c86-4569-b49d-606acafa57ee-kube-api-access-nfnvz\") pod \"maas-api-revoke-sub-79db70ec-9mdvf\" (UID: \"dd2068b1-9c86-4569-b49d-606acafa57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:32.107218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.107164    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79db70ec-9mdvf\" (UID: \"dd2068b1-9c86-4569-b49d-606acafa57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:32.107816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.107307    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79db70ec-9mdvf\" (UID: \"dd2068b1-9c86-4569-b49d-606acafa57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:32.208721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.208689    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfnvz\" (UniqueName: \"kubernetes.io/projected/dd2068b1-9c86-4569-b49d-606acafa57ee-kube-api-access-nfnvz\") pod \"maas-api-revoke-sub-79db70ec-9mdvf\" (UID: \"dd2068b1-9c86-4569-b49d-606acafa57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:32.208903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.208769    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79db70ec-9mdvf\" (UID: \"dd2068b1-9c86-4569-b49d-606acafa57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:32.208903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.208824    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79db70ec-9mdvf\" (UID: \"dd2068b1-9c86-4569-b49d-606acafa57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:32.209560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.209536    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79db70ec-9mdvf\" (UID: \"dd2068b1-9c86-4569-b49d-606acafa57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:32.211111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.211088    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79db70ec-9mdvf\" (UID: \"dd2068b1-9c86-4569-b49d-606acafa57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:32.229371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.229345    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfnvz\" (UniqueName: \"kubernetes.io/projected/dd2068b1-9c86-4569-b49d-606acafa57ee-kube-api-access-nfnvz\") pod \"maas-api-revoke-sub-79db70ec-9mdvf\" (UID: \"dd2068b1-9c86-4569-b49d-606acafa57ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:32.254617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.254587    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:32.364505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.364414    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:32.673181 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:32.673153    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2068b1_9c86_4569_b49d_606acafa57ee.slice/crio-445611eb29ceb1e80362fab33eba6cda068e218c15cfaae46e7b473218fe17c9 WatchSource:0}: Error finding container 445611eb29ceb1e80362fab33eba6cda068e218c15cfaae46e7b473218fe17c9: Status 404 returned error can't find the container with id 445611eb29ceb1e80362fab33eba6cda068e218c15cfaae46e7b473218fe17c9
Oct 29 22:31:32.679140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:32.679113    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"]
Oct 29 22:31:33.184812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.184770    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"]
Oct 29 22:31:33.189143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.189127    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:33.207884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.207854    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"]
Oct 29 22:31:33.222247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.222220    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e4374a0-lkg4p\" (UID: \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:33.222450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.222427    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4wxm\" (UniqueName: \"kubernetes.io/projected/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-kube-api-access-l4wxm\") pod \"maas-api-revoke-sub-3e4374a0-lkg4p\" (UID: \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:33.222612 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.222589    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e4374a0-lkg4p\" (UID: \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:33.323768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.323692    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4wxm\" (UniqueName: \"kubernetes.io/projected/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-kube-api-access-l4wxm\") pod \"maas-api-revoke-sub-3e4374a0-lkg4p\" (UID: \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:33.323934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.323789    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e4374a0-lkg4p\" (UID: \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:33.323934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.323842    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e4374a0-lkg4p\" (UID: \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:33.324462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.324435    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e4374a0-lkg4p\" (UID: \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:33.326235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.326211    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e4374a0-lkg4p\" (UID: \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:33.345355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.345334    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4wxm\" (UniqueName: \"kubernetes.io/projected/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-kube-api-access-l4wxm\") pod \"maas-api-revoke-sub-3e4374a0-lkg4p\" (UID: \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:33.503118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.503085    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:33.656400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.656279    4860 generic.go:356] "Generic (PLEG): container finished" podID="dd2068b1-9c86-4569-b49d-606acafa57ee" containerID="4d15d62203cca57986eaf806f6f801c26019c3d352241cdeda3f79ae76257e99" exitCode=0
Oct 29 22:31:33.656400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.656372    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf" event={"ID":"dd2068b1-9c86-4569-b49d-606acafa57ee","Type":"ContainerDied","Data":"4d15d62203cca57986eaf806f6f801c26019c3d352241cdeda3f79ae76257e99"}
Oct 29 22:31:33.656616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.656423    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf" event={"ID":"dd2068b1-9c86-4569-b49d-606acafa57ee","Type":"ContainerStarted","Data":"445611eb29ceb1e80362fab33eba6cda068e218c15cfaae46e7b473218fe17c9"}
Oct 29 22:31:33.805631 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:33.805585    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78a9eb3_aeb2_40d5_8a5c_31be12e6b380.slice/crio-d697942f5dc0332c9c665eaf472230334b54c5fcb2f7fc790889168442cab371 WatchSource:0}: Error finding container d697942f5dc0332c9c665eaf472230334b54c5fcb2f7fc790889168442cab371: Status 404 returned error can't find the container with id d697942f5dc0332c9c665eaf472230334b54c5fcb2f7fc790889168442cab371
Oct 29 22:31:33.809340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.809303    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"]
Oct 29 22:31:33.852033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:33.852007    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:34.666823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.666798    4860 generic.go:356] "Generic (PLEG): container finished" podID="d78a9eb3-aeb2-40d5-8a5c-31be12e6b380" containerID="83ad9f7b7e9644e62e03921c880f5e081d1a1c33fd2c0119eaf2974f90e0b4ab" exitCode=0
Oct 29 22:31:34.667177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.666877    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p" event={"ID":"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380","Type":"ContainerDied","Data":"83ad9f7b7e9644e62e03921c880f5e081d1a1c33fd2c0119eaf2974f90e0b4ab"}
Oct 29 22:31:34.667177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.666930    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p" event={"ID":"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380","Type":"ContainerStarted","Data":"d697942f5dc0332c9c665eaf472230334b54c5fcb2f7fc790889168442cab371"}
Oct 29 22:31:34.802515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.802478    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"]
Oct 29 22:31:34.807898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.807878    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:34.824429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.824402    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"]
Oct 29 22:31:34.844703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.844656    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk6wq\" (UniqueName: \"kubernetes.io/projected/924a9308-baf9-4f32-aa16-ba545a327ee5-kube-api-access-fk6wq\") pod \"maas-api-revoke-sub-6fd631cb-6sp22\" (UID: \"924a9308-baf9-4f32-aa16-ba545a327ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:34.844816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.844714    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fd631cb-6sp22\" (UID: \"924a9308-baf9-4f32-aa16-ba545a327ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:34.844953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.844932    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fd631cb-6sp22\" (UID: \"924a9308-baf9-4f32-aa16-ba545a327ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:34.857537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.857516    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:34.946448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.946415    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2068b1-9c86-4569-b49d-606acafa57ee-kube-api-access-nfnvz\" (UniqueName: \"kubernetes.io/projected/dd2068b1-9c86-4569-b49d-606acafa57ee-kube-api-access-nfnvz\") pod \"dd2068b1-9c86-4569-b49d-606acafa57ee\" (UID: \"dd2068b1-9c86-4569-b49d-606acafa57ee\") "
Oct 29 22:31:34.946607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.946579    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-xks-ca\") pod \"dd2068b1-9c86-4569-b49d-606acafa57ee\" (UID: \"dd2068b1-9c86-4569-b49d-606acafa57ee\") "
Oct 29 22:31:34.946660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.946621    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-openshift-ca\") pod \"dd2068b1-9c86-4569-b49d-606acafa57ee\" (UID: \"dd2068b1-9c86-4569-b49d-606acafa57ee\") "
Oct 29 22:31:34.946876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.946854    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk6wq\" (UniqueName: \"kubernetes.io/projected/924a9308-baf9-4f32-aa16-ba545a327ee5-kube-api-access-fk6wq\") pod \"maas-api-revoke-sub-6fd631cb-6sp22\" (UID: \"924a9308-baf9-4f32-aa16-ba545a327ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:34.947002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.946884    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fd631cb-6sp22\" (UID: \"924a9308-baf9-4f32-aa16-ba545a327ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:34.947083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.947028    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-openshift-ca" pod "dd2068b1-9c86-4569-b49d-606acafa57ee" (UID: "dd2068b1-9c86-4569-b49d-606acafa57ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:31:34.947083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.947065    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fd631cb-6sp22\" (UID: \"924a9308-baf9-4f32-aa16-ba545a327ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:34.947194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.947150    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:34.947848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.947823    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fd631cb-6sp22\" (UID: \"924a9308-baf9-4f32-aa16-ba545a327ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:34.948748 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.948726    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2068b1-9c86-4569-b49d-606acafa57ee-kube-api-access-nfnvz" pod "dd2068b1-9c86-4569-b49d-606acafa57ee" (UID: "dd2068b1-9c86-4569-b49d-606acafa57ee"). InnerVolumeSpecName "kube-api-access-nfnvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:31:34.948822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.948751    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-xks-ca" pod "dd2068b1-9c86-4569-b49d-606acafa57ee" (UID: "dd2068b1-9c86-4569-b49d-606acafa57ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:31:34.949502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.949483    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fd631cb-6sp22\" (UID: \"924a9308-baf9-4f32-aa16-ba545a327ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:34.967480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:34.967455    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk6wq\" (UniqueName: \"kubernetes.io/projected/924a9308-baf9-4f32-aa16-ba545a327ee5-kube-api-access-fk6wq\") pod \"maas-api-revoke-sub-6fd631cb-6sp22\" (UID: \"924a9308-baf9-4f32-aa16-ba545a327ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:35.047984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.047953    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd2068b1-9c86-4569-b49d-606acafa57ee-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:35.047984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.047981    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfnvz\" (UniqueName: \"kubernetes.io/projected/dd2068b1-9c86-4569-b49d-606acafa57ee-kube-api-access-nfnvz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:35.155174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.155104    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:35.452029 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:35.451999    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924a9308_baf9_4f32_aa16_ba545a327ee5.slice/crio-5d544c4e1548a2e6de5578871fe41c3e73ec9cc7d5146c76a88695ce90099bb7 WatchSource:0}: Error finding container 5d544c4e1548a2e6de5578871fe41c3e73ec9cc7d5146c76a88695ce90099bb7: Status 404 returned error can't find the container with id 5d544c4e1548a2e6de5578871fe41c3e73ec9cc7d5146c76a88695ce90099bb7
Oct 29 22:31:35.456551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.456530    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"]
Oct 29 22:31:35.540356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.540289    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:35.676860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.676831    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"
Oct 29 22:31:35.676860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.676854    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf" event={"ID":"dd2068b1-9c86-4569-b49d-606acafa57ee","Type":"ContainerDied","Data":"445611eb29ceb1e80362fab33eba6cda068e218c15cfaae46e7b473218fe17c9"}
Oct 29 22:31:35.677314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.676905    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="445611eb29ceb1e80362fab33eba6cda068e218c15cfaae46e7b473218fe17c9"
Oct 29 22:31:35.678192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.678163    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22" event={"ID":"924a9308-baf9-4f32-aa16-ba545a327ee5","Type":"ContainerStarted","Data":"5d544c4e1548a2e6de5578871fe41c3e73ec9cc7d5146c76a88695ce90099bb7"}
Oct 29 22:31:35.849483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.849461    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:35.960101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.960066    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-openshift-ca\") pod \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\" (UID: \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\") "
Oct 29 22:31:35.960259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.960145    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-xks-ca\") pod \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\" (UID: \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\") "
Oct 29 22:31:35.960301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.960256    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-kube-api-access-l4wxm\" (UniqueName: \"kubernetes.io/projected/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-kube-api-access-l4wxm\") pod \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\" (UID: \"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380\") "
Oct 29 22:31:35.960547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.960516    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-openshift-ca" pod "d78a9eb3-aeb2-40d5-8a5c-31be12e6b380" (UID: "d78a9eb3-aeb2-40d5-8a5c-31be12e6b380"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:31:35.960948 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.960925    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:35.962524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.962498    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-xks-ca" pod "d78a9eb3-aeb2-40d5-8a5c-31be12e6b380" (UID: "d78a9eb3-aeb2-40d5-8a5c-31be12e6b380"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:31:35.962596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:35.962521    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-kube-api-access-l4wxm" pod "d78a9eb3-aeb2-40d5-8a5c-31be12e6b380" (UID: "d78a9eb3-aeb2-40d5-8a5c-31be12e6b380"). InnerVolumeSpecName "kube-api-access-l4wxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:31:36.062433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.062381    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:36.062433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.062403    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4wxm\" (UniqueName: \"kubernetes.io/projected/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380-kube-api-access-l4wxm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:36.117503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.117475    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"]
Oct 29 22:31:36.118085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.118072    4860 cpu_manager.go:414] "containerMap: removing container" podUID="dd2068b1-9c86-4569-b49d-606acafa57ee" containerName="revoke-keys"
Oct 29 22:31:36.118131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.118087    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2068b1-9c86-4569-b49d-606acafa57ee" containerName="revoke-keys"
Oct 29 22:31:36.118131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.118097    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d78a9eb3-aeb2-40d5-8a5c-31be12e6b380" containerName="revoke-keys"
Oct 29 22:31:36.118131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.118102    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78a9eb3-aeb2-40d5-8a5c-31be12e6b380" containerName="revoke-keys"
Oct 29 22:31:36.118225 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.118216    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2068b1-9c86-4569-b49d-606acafa57ee" containerName="revoke-keys"
Oct 29 22:31:36.118260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.118226    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78a9eb3-aeb2-40d5-8a5c-31be12e6b380" containerName="revoke-keys"
Oct 29 22:31:36.121963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.121946    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.149753 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.149609    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"]
Oct 29 22:31:36.163759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.163734    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.163878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.163830    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.163878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.163870    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a262353-8924-4f7c-94c5-eebde78895aa-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.163995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.163980    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a262353-8924-4f7c-94c5-eebde78895aa-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.164033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.164012    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.164114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.164098    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chzb5\" (UniqueName: \"kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-kube-api-access-chzb5\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.164164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.164152    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.164235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.164221    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.164346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.164313    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.265386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.265355    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chzb5\" (UniqueName: \"kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-kube-api-access-chzb5\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.265507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.265422    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.265507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.265438    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.265507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.265464    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.265678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.265506    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.265678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.265563    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.265678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.265589    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a262353-8924-4f7c-94c5-eebde78895aa-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.265678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.265662    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a262353-8924-4f7c-94c5-eebde78895aa-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.265937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.265705    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.266000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.265956    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-certs\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.266130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.266105    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-data\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.266174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.266131    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-credential-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.266240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.266220    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-socket\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.266403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.266387    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a262353-8924-4f7c-94c5-eebde78895aa-istiod-ca-cert\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.267986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.267963    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-envoy\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.268151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.268134    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a262353-8924-4f7c-94c5-eebde78895aa-istio-podinfo\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.285945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.285918    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-istio-token\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.286038 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.286020    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chzb5\" (UniqueName: \"kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-kube-api-access-chzb5\") pod \"e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") " pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.436639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.436613    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:36.687995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.687916    4860 generic.go:356] "Generic (PLEG): container finished" podID="924a9308-baf9-4f32-aa16-ba545a327ee5" containerID="99cda612383bf7dbb089fde1a73245174e6143f8b09d91f55577eb377df999c6" exitCode=0
Oct 29 22:31:36.688407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.688005    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22" event={"ID":"924a9308-baf9-4f32-aa16-ba545a327ee5","Type":"ContainerDied","Data":"99cda612383bf7dbb089fde1a73245174e6143f8b09d91f55577eb377df999c6"}
Oct 29 22:31:36.689518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.689493    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p" event={"ID":"d78a9eb3-aeb2-40d5-8a5c-31be12e6b380","Type":"ContainerDied","Data":"d697942f5dc0332c9c665eaf472230334b54c5fcb2f7fc790889168442cab371"}
Oct 29 22:31:36.689618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.689537    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d697942f5dc0332c9c665eaf472230334b54c5fcb2f7fc790889168442cab371"
Oct 29 22:31:36.689618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.689499    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"
Oct 29 22:31:36.752436 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:36.752410    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a262353_8924_4f7c_94c5_eebde78895aa.slice/crio-f3e2ff7edb3800222d469730a6c6c480efb64775f35cc2edd39ccc2b54fd6ea8 WatchSource:0}: Error finding container f3e2ff7edb3800222d469730a6c6c480efb64775f35cc2edd39ccc2b54fd6ea8: Status 404 returned error can't find the container with id f3e2ff7edb3800222d469730a6c6c480efb64775f35cc2edd39ccc2b54fd6ea8
Oct 29 22:31:36.755003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.754964    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:36.755077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.755049    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:36.755111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.755088    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:36.757609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:36.757585    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"]
Oct 29 22:31:37.151273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:37.151197    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:37.701660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:37.701621    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" event={"ID":"6a262353-8924-4f7c-94c5-eebde78895aa","Type":"ContainerStarted","Data":"5ab721d8b6f631a3947b2e0731cb3d529b2b066a381e98e2566c50fc8dbce903"}
Oct 29 22:31:37.702140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:37.701685    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" event={"ID":"6a262353-8924-4f7c-94c5-eebde78895aa","Type":"ContainerStarted","Data":"f3e2ff7edb3800222d469730a6c6c480efb64775f35cc2edd39ccc2b54fd6ea8"}
Oct 29 22:31:37.805950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:37.802256    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" podStartSLOduration=1.8022369889999998 podStartE2EDuration="1.802236989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:37.759893312 +0000 UTC m=+1039.784669469" watchObservedRunningTime="2026-10-29 22:31:37.802236989 +0000 UTC m=+1039.827013123"
Oct 29 22:31:37.902098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:37.902070    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:38.003723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.003632    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-openshift-ca\") pod \"924a9308-baf9-4f32-aa16-ba545a327ee5\" (UID: \"924a9308-baf9-4f32-aa16-ba545a327ee5\") "
Oct 29 22:31:38.003873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.003819    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-xks-ca\") pod \"924a9308-baf9-4f32-aa16-ba545a327ee5\" (UID: \"924a9308-baf9-4f32-aa16-ba545a327ee5\") "
Oct 29 22:31:38.003966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.003950    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924a9308-baf9-4f32-aa16-ba545a327ee5-kube-api-access-fk6wq\" (UniqueName: \"kubernetes.io/projected/924a9308-baf9-4f32-aa16-ba545a327ee5-kube-api-access-fk6wq\") pod \"924a9308-baf9-4f32-aa16-ba545a327ee5\" (UID: \"924a9308-baf9-4f32-aa16-ba545a327ee5\") "
Oct 29 22:31:38.004026 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.003998    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-openshift-ca" pod "924a9308-baf9-4f32-aa16-ba545a327ee5" (UID: "924a9308-baf9-4f32-aa16-ba545a327ee5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:31:38.004534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.004506    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:38.007270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.006270    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924a9308-baf9-4f32-aa16-ba545a327ee5-kube-api-access-fk6wq" pod "924a9308-baf9-4f32-aa16-ba545a327ee5" (UID: "924a9308-baf9-4f32-aa16-ba545a327ee5"). InnerVolumeSpecName "kube-api-access-fk6wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:31:38.007270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.006719    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-xks-ca" pod "924a9308-baf9-4f32-aa16-ba545a327ee5" (UID: "924a9308-baf9-4f32-aa16-ba545a327ee5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:31:38.105656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.105617    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924a9308-baf9-4f32-aa16-ba545a327ee5-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:38.105656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.105652    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk6wq\" (UniqueName: \"kubernetes.io/projected/924a9308-baf9-4f32-aa16-ba545a327ee5-kube-api-access-fk6wq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:38.437222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.437185    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:38.439085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.439060    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 29 22:31:38.439201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.439118    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 29 22:31:38.711131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.711046    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"
Oct 29 22:31:38.711131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.711053    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22" event={"ID":"924a9308-baf9-4f32-aa16-ba545a327ee5","Type":"ContainerDied","Data":"5d544c4e1548a2e6de5578871fe41c3e73ec9cc7d5146c76a88695ce90099bb7"}
Oct 29 22:31:38.711131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.711114    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d544c4e1548a2e6de5578871fe41c3e73ec9cc7d5146c76a88695ce90099bb7"
Oct 29 22:31:38.893765 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:38.893717    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:39.437224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:39.437189    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 29 22:31:39.437421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:39.437244    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 29 22:31:40.437509 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:40.437467    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 29 22:31:40.437935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:40.437532    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 29 22:31:40.557633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:40.557599    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:41.437795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.437764    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 29 22:31:41.438202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.437817    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 29 22:31:41.860120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.860045    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"]
Oct 29 22:31:41.860711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.860695    4860 cpu_manager.go:414] "containerMap: removing container" podUID="924a9308-baf9-4f32-aa16-ba545a327ee5" containerName="revoke-keys"
Oct 29 22:31:41.860764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.860713    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924a9308-baf9-4f32-aa16-ba545a327ee5" containerName="revoke-keys"
Oct 29 22:31:41.860859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.860849    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="924a9308-baf9-4f32-aa16-ba545a327ee5" containerName="revoke-keys"
Oct 29 22:31:41.864664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.864649    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:41.891709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.891676    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:31:41.891944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.891928    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:31:41.892163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.892148    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:31:41.903438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.900688    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"]
Oct 29 22:31:41.904393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.903873    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:31:41.953297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.953270    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-813dc797-hsvkp\" (UID: \"07177684-d8bb-49e3-9634-77c995f3840f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:41.953435 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.953308    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jgk5\" (UniqueName: \"kubernetes.io/projected/07177684-d8bb-49e3-9634-77c995f3840f-kube-api-access-8jgk5\") pod \"maas-api-revoke-sub-813dc797-hsvkp\" (UID: \"07177684-d8bb-49e3-9634-77c995f3840f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:41.953584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:41.953565    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-813dc797-hsvkp\" (UID: \"07177684-d8bb-49e3-9634-77c995f3840f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:42.054632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:42.054599    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-813dc797-hsvkp\" (UID: \"07177684-d8bb-49e3-9634-77c995f3840f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:42.054812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:42.054640    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jgk5\" (UniqueName: \"kubernetes.io/projected/07177684-d8bb-49e3-9634-77c995f3840f-kube-api-access-8jgk5\") pod \"maas-api-revoke-sub-813dc797-hsvkp\" (UID: \"07177684-d8bb-49e3-9634-77c995f3840f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:42.054812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:42.054779    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-813dc797-hsvkp\" (UID: \"07177684-d8bb-49e3-9634-77c995f3840f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:42.055451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:42.055424    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-813dc797-hsvkp\" (UID: \"07177684-d8bb-49e3-9634-77c995f3840f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:42.057155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:42.057129    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-813dc797-hsvkp\" (UID: \"07177684-d8bb-49e3-9634-77c995f3840f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:42.088345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:42.088216    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jgk5\" (UniqueName: \"kubernetes.io/projected/07177684-d8bb-49e3-9634-77c995f3840f-kube-api-access-8jgk5\") pod \"maas-api-revoke-sub-813dc797-hsvkp\" (UID: \"07177684-d8bb-49e3-9634-77c995f3840f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:42.218890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:42.218858    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:42.256082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:42.256055    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:42.437187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:42.437157    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 29 22:31:42.437436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:42.437211    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 29 22:31:42.540306 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:42.540275    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07177684_d8bb_49e3_9634_77c995f3840f.slice/crio-28cecb82c3fbb75d4ffd974097b6fb3852f75f8f451709a6eb97f8cd5143c3e7 WatchSource:0}: Error finding container 28cecb82c3fbb75d4ffd974097b6fb3852f75f8f451709a6eb97f8cd5143c3e7: Status 404 returned error can't find the container with id 28cecb82c3fbb75d4ffd974097b6fb3852f75f8f451709a6eb97f8cd5143c3e7
Oct 29 22:31:42.543749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:42.543721    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"]
Oct 29 22:31:42.745399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:42.745364    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp" event={"ID":"07177684-d8bb-49e3-9634-77c995f3840f","Type":"ContainerStarted","Data":"28cecb82c3fbb75d4ffd974097b6fb3852f75f8f451709a6eb97f8cd5143c3e7"}
Oct 29 22:31:43.437807 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.437766    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 29 22:31:43.437958 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.437836    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 29 22:31:43.580928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.580882    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"]
Oct 29 22:31:43.586198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.586177    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:43.703784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.691422    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"]
Oct 29 22:31:43.704029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.699383    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b97c28ca-h9tz6\" (UID: \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:43.704430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.704408    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkhxs\" (UniqueName: \"kubernetes.io/projected/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-kube-api-access-kkhxs\") pod \"maas-api-revoke-sub-b97c28ca-h9tz6\" (UID: \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:43.704575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.704560    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b97c28ca-h9tz6\" (UID: \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:43.760007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.759964    4860 generic.go:356] "Generic (PLEG): container finished" podID="07177684-d8bb-49e3-9634-77c995f3840f" containerID="8c38771ebf9d3b36cf97cf1576940d2f9d4c6fd3198430463f2e14b80145cc62" exitCode=0
Oct 29 22:31:43.760314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.760291    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp" event={"ID":"07177684-d8bb-49e3-9634-77c995f3840f","Type":"ContainerDied","Data":"8c38771ebf9d3b36cf97cf1576940d2f9d4c6fd3198430463f2e14b80145cc62"}
Oct 29 22:31:43.812345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.807904    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhxs\" (UniqueName: \"kubernetes.io/projected/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-kube-api-access-kkhxs\") pod \"maas-api-revoke-sub-b97c28ca-h9tz6\" (UID: \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:43.812345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.807985    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b97c28ca-h9tz6\" (UID: \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:43.812345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.808086    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b97c28ca-h9tz6\" (UID: \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:43.812345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.809213    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b97c28ca-h9tz6\" (UID: \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:43.859791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.826020    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b97c28ca-h9tz6\" (UID: \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:43.859791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.837565    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhxs\" (UniqueName: \"kubernetes.io/projected/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-kube-api-access-kkhxs\") pod \"maas-api-revoke-sub-b97c28ca-h9tz6\" (UID: \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:43.930362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:43.928033    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:44.125363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:44.110302    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:45.132353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.128646    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:31:45.132353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.128723    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:45.132353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.130477    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:31:45.132929 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.130692    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:45.132929 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.131663    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:45.132929 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.132900    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:45.457254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.452556    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:45.457254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.452625    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:45.581351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.580388    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:45.630557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.610646    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"]
Oct 29 22:31:45.762379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.761430    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:45.810350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.809409    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:45.874411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.874373    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:45.997547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:45.997484    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:31:46.026267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.023392    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.181631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.130017    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:46.181631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.130076    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" podUID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.181631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.132150    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.181631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.132510    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.181631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.132802    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:31:46.181631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.132853    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:31:46.181631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.133184    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.308757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.230966    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:31:46.308757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.302395    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:31:46.308757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.302736    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:31:46.308757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.303000    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:31:46.364350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.357470    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.518446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.493402    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:46.518446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.493474    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.637413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.630294    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Oct 29 22:31:46.709348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.705893    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:46.709348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.705965    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.778355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.778288    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.788527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.788498    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:31:46.788766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.788748    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:31:46.788913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.788901    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:31:46.813904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.813877    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:31:46.873350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.818444    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.873350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.818828    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:31:46.877385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.875414    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Oct 29 22:31:46.884356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.877578    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:46.884562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.884530    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:46.893471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:46.884598    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:31:47.059357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.057683    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.059357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.058026    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.059357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.058351    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:31:47.059357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.058415    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:31:47.172454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.171520    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:31:47.172454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.171867    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.172454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.172111    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:31:47.333913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.333853    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:31:47.340365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.334875    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:31:47.340365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.335204    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l6lbv" podUID="0e42bac5-f252-4c5f-9739-f0b300489d73" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.403359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.344476    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.440576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.440533    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6" event={"ID":"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63","Type":"ContainerStarted","Data":"e97d337f64d9ceb44b5de6b8e352a6205005cc840d740b14ad22c32c847e85be"}
Oct 29 22:31:47.532233 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.532199    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:47.570916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.570790    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:47.570916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.570848    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.571597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.571228    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:47.571597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.571272    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.750399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.618710    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07177684-d8bb-49e3-9634-77c995f3840f-kube-api-access-8jgk5\" (UniqueName: \"kubernetes.io/projected/07177684-d8bb-49e3-9634-77c995f3840f-kube-api-access-8jgk5\") pod \"07177684-d8bb-49e3-9634-77c995f3840f\" (UID: \"07177684-d8bb-49e3-9634-77c995f3840f\") "
Oct 29 22:31:47.814390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.794558    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-openshift-ca\") pod \"07177684-d8bb-49e3-9634-77c995f3840f\" (UID: \"07177684-d8bb-49e3-9634-77c995f3840f\") "
Oct 29 22:31:47.814390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.794634    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-xks-ca\") pod \"07177684-d8bb-49e3-9634-77c995f3840f\" (UID: \"07177684-d8bb-49e3-9634-77c995f3840f\") "
Oct 29 22:31:47.814390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.794465    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07177684-d8bb-49e3-9634-77c995f3840f-kube-api-access-8jgk5" pod "07177684-d8bb-49e3-9634-77c995f3840f" (UID: "07177684-d8bb-49e3-9634-77c995f3840f"). InnerVolumeSpecName "kube-api-access-8jgk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:31:47.814390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.695955    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:47.814390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.793530    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.833210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.832040    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:31:47.884446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.881912    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:47.914851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.914821    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jgk5\" (UniqueName: \"kubernetes.io/projected/07177684-d8bb-49e3-9634-77c995f3840f-kube-api-access-8jgk5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:47.920531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.920496    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-xks-ca" pod "07177684-d8bb-49e3-9634-77c995f3840f" (UID: "07177684-d8bb-49e3-9634-77c995f3840f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:31:47.949354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:47.947555    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-openshift-ca" pod "07177684-d8bb-49e3-9634-77c995f3840f" (UID: "07177684-d8bb-49e3-9634-77c995f3840f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:31:48.027104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.027014    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:48.052047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.052018    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07177684-d8bb-49e3-9634-77c995f3840f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:48.064061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.064005    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.064760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.064715    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:31:48.218145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.194779    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:31:48.260404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.225409    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:31:48.260404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.225773    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:31:48.342533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.342428    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.343457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.343422    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:31:48.371167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.351465    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.785788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.785732    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:48.851200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.851160    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:31:48.937490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.937436    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:31:48.937801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.937754    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.004521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:48.999485    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.008077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.008041    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:31:49.092555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.079776    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:31:49.147580 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:31:49.147543    4860 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.015s"
Oct 29 22:31:49.147832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.147807    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:31:49.148015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.147996    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:31:49.148168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.148156    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:31:49.148299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.148290    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:31:49.194971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.193554    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:31:49.211358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.211308    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:31:49.220917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.220888    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp" event={"ID":"07177684-d8bb-49e3-9634-77c995f3840f","Type":"ContainerDied","Data":"28cecb82c3fbb75d4ffd974097b6fb3852f75f8f451709a6eb97f8cd5143c3e7"}
Oct 29 22:31:49.221126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.221110    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28cecb82c3fbb75d4ffd974097b6fb3852f75f8f451709a6eb97f8cd5143c3e7"
Oct 29 22:31:49.221245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.214831    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"
Oct 29 22:31:49.260534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.260512    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:31:49.277977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.277953    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:31:49.279774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.279758    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:31:49.280453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.280439    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:31:49.280641 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:31:49.280622    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:31:49.281287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.281266    4860 generic.go:356] "Generic (PLEG): container finished" podID="ec5d62ba-ce99-47ca-83c3-82bacd5b4d63" containerID="da10ffcd693f5d618bf1ef3197d2cc4d960691428b56a2295ff8dc9b6aa07845" exitCode=0
Oct 29 22:31:49.282662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.282617    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6" event={"ID":"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63","Type":"ContainerDied","Data":"da10ffcd693f5d618bf1ef3197d2cc4d960691428b56a2295ff8dc9b6aa07845"}
Oct 29 22:31:49.304004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.303978    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:49.304936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.304918    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:31:49.428312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.428232    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:49.464871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.464825    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:49.466061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.466036    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:49.467223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.467198    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:31:49.623206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:49.623178    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:50.454231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.454207    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:50.490583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.490558    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-openshift-ca\") pod \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\" (UID: \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\") "
Oct 29 22:31:50.490729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.490614    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-xks-ca\") pod \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\" (UID: \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\") "
Oct 29 22:31:50.490785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.490770    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-kube-api-access-kkhxs\" (UniqueName: \"kubernetes.io/projected/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-kube-api-access-kkhxs\") pod \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\" (UID: \"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63\") "
Oct 29 22:31:50.490895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.490874    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-openshift-ca" pod "ec5d62ba-ce99-47ca-83c3-82bacd5b4d63" (UID: "ec5d62ba-ce99-47ca-83c3-82bacd5b4d63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:31:50.491206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.491189    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:50.492812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.492782    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-xks-ca" pod "ec5d62ba-ce99-47ca-83c3-82bacd5b4d63" (UID: "ec5d62ba-ce99-47ca-83c3-82bacd5b4d63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:31:50.492901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.492813    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-kube-api-access-kkhxs" pod "ec5d62ba-ce99-47ca-83c3-82bacd5b4d63" (UID: "ec5d62ba-ce99-47ca-83c3-82bacd5b4d63"). InnerVolumeSpecName "kube-api-access-kkhxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:31:50.592279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.592207    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:50.592279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.592234    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkhxs\" (UniqueName: \"kubernetes.io/projected/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63-kube-api-access-kkhxs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:31:50.602938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.602905    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"]
Oct 29 22:31:50.603623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.603603    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ec5d62ba-ce99-47ca-83c3-82bacd5b4d63" containerName="revoke-keys"
Oct 29 22:31:50.603623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.603623    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec5d62ba-ce99-47ca-83c3-82bacd5b4d63" containerName="revoke-keys"
Oct 29 22:31:50.603786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.603670    4860 cpu_manager.go:414] "containerMap: removing container" podUID="07177684-d8bb-49e3-9634-77c995f3840f" containerName="revoke-keys"
Oct 29 22:31:50.603786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.603678    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07177684-d8bb-49e3-9634-77c995f3840f" containerName="revoke-keys"
Oct 29 22:31:50.603880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.603864    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec5d62ba-ce99-47ca-83c3-82bacd5b4d63" containerName="revoke-keys"
Oct 29 22:31:50.603930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.603886    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="07177684-d8bb-49e3-9634-77c995f3840f" containerName="revoke-keys"
Oct 29 22:31:50.607653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.607633    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.633506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.633484    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"]
Oct 29 22:31:50.693146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.693113    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.693309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.693158    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.693309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.693183    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77e610e-005e-4e06-a301-161d60b6df15-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.693309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.693290    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.693493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.693400    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.693550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.693488    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.693550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.693528    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77e610e-005e-4e06-a301-161d60b6df15-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.693714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.693593    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.693714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.693640    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkprn\" (UniqueName: \"kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-kube-api-access-zkprn\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.794437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.794411    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.794540 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.794449    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkprn\" (UniqueName: \"kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-kube-api-access-zkprn\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.794540 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.794533    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.794619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.794560    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.794619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.794588    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77e610e-005e-4e06-a301-161d60b6df15-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.794619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.794604    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.794742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.794654    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.794799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.794737    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.794799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.794782    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77e610e-005e-4e06-a301-161d60b6df15-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.794897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.794811    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.795018 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.794993    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-data\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.795081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.795019    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-certs\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.795169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.795143    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-credential-socket\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.795647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.795624    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77e610e-005e-4e06-a301-161d60b6df15-istiod-ca-cert\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.796978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.796954    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-envoy\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.797214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.797192    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77e610e-005e-4e06-a301-161d60b6df15-istio-podinfo\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.816989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.816964    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkprn\" (UniqueName: \"kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-kube-api-access-zkprn\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.818111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.818091    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-istio-token\") pod \"e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") " pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:50.881597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.881544    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"]
Oct 29 22:31:50.887637 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.887619    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:50.897356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.897316    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:50.897506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.897482    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:50.899093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.898427    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:50.899093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.898466    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:50.899093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.898512    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:50.899093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.898536    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvszx\" (UniqueName: \"kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-kube-api-access-fvszx\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:50.899093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.898591    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:50.899093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.898655    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:50.899093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.898696    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb026b1-e8ce-461a-bcdc-a38955de1842-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:50.908340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.908308    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"]
Oct 29 22:31:50.922203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:50.922173    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:51.000568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.000527    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.000680 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.000600    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb026b1-e8ce-461a-bcdc-a38955de1842-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.000680 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.000661    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.000752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.000727    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.001571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.000861    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.001571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.000881    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-data\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.001571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.000892    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.001571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.000950    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.001571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.000967    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvszx\" (UniqueName: \"kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-kube-api-access-fvszx\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.001571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.001014    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.001571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.001119    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-credential-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.001571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.001244    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-socket\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.001571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.001468    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-certs\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.001571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.001530    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb026b1-e8ce-461a-bcdc-a38955de1842-istiod-ca-cert\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.003657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.003617    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-envoy\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.004003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.003980    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-podinfo\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.023385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.023356    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-token\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.024678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.024652    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvszx\" (UniqueName: \"kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-kube-api-access-fvszx\") pod \"e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") " pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.208672 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.208642    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:51.236762 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:51.236733    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77e610e_005e_4e06_a301_161d60b6df15.slice/crio-63115ed852f7a05dd3fc3de1cf7fbf3be77696a99c19a0162db3d5188bb2f2fe WatchSource:0}: Error finding container 63115ed852f7a05dd3fc3de1cf7fbf3be77696a99c19a0162db3d5188bb2f2fe: Status 404 returned error can't find the container with id 63115ed852f7a05dd3fc3de1cf7fbf3be77696a99c19a0162db3d5188bb2f2fe
Oct 29 22:31:51.239406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.239369    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:51.239501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.239438    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:51.239501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.239481    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:51.240838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.240815    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"]
Oct 29 22:31:51.305121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.305089    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:51.308008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.307985    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"
Oct 29 22:31:51.308140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.307988    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6" event={"ID":"ec5d62ba-ce99-47ca-83c3-82bacd5b4d63","Type":"ContainerDied","Data":"e97d337f64d9ceb44b5de6b8e352a6205005cc840d740b14ad22c32c847e85be"}
Oct 29 22:31:51.308140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.308116    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e97d337f64d9ceb44b5de6b8e352a6205005cc840d740b14ad22c32c847e85be"
Oct 29 22:31:51.309625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.309599    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np" event={"ID":"e77e610e-005e-4e06-a301-161d60b6df15","Type":"ContainerStarted","Data":"63115ed852f7a05dd3fc3de1cf7fbf3be77696a99c19a0162db3d5188bb2f2fe"}
Oct 29 22:31:51.532559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.532506    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:51.533079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.532611    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:51.533079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.532660    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:31:51.533220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:51.533141    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"]
Oct 29 22:31:52.339925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:52.339204    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np" event={"ID":"e77e610e-005e-4e06-a301-161d60b6df15","Type":"ContainerStarted","Data":"a93a372e64e09cb38577dcbb31f4d7b22f1089a14c885bd82ca161222083d1e9"}
Oct 29 22:31:52.341534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:52.341505    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w" event={"ID":"1bb026b1-e8ce-461a-bcdc-a38955de1842","Type":"ContainerStarted","Data":"0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e"}
Oct 29 22:31:52.341661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:52.341550    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w" event={"ID":"1bb026b1-e8ce-461a-bcdc-a38955de1842","Type":"ContainerStarted","Data":"f46bfaa13c9be86bf3a10f2fc6ba6a59afc9a93a3decf49dc26f16b3eb6b0e44"}
Oct 29 22:31:52.478369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:52.478289    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np" podStartSLOduration=2.478269937 podStartE2EDuration="2.478269937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:52.431341864 +0000 UTC m=+1054.456118001" watchObservedRunningTime="2026-10-29 22:31:52.478269937 +0000 UTC m=+1054.503046071"
Oct 29 22:31:52.498388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:52.480251    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w" podStartSLOduration=2.4802405370000002 podStartE2EDuration="2.480240537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:52.477807515 +0000 UTC m=+1054.502583684" watchObservedRunningTime="2026-10-29 22:31:52.480240537 +0000 UTC m=+1054.505016669"
Oct 29 22:31:52.922792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:52.922752    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:52.924535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:52.924507    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:31:52.924663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:52.924563    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np" podUID="e77e610e-005e-4e06-a301-161d60b6df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 29 22:31:53.212348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:53.211495    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:53.212348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:53.211615    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:31:53.212348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:53.211658    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w" podUID="1bb026b1-e8ce-461a-bcdc-a38955de1842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 29 22:31:53.263585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:53.263554    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:53.922913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:53.922862    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:31:53.923441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:53.922928    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np" podUID="e77e610e-005e-4e06-a301-161d60b6df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 29 22:31:54.217686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:54.216825    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:31:54.217686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:54.216889    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w" podUID="1bb026b1-e8ce-461a-bcdc-a38955de1842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 29 22:31:54.886358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:54.885663    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:54.925389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:54.922685    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:31:54.925389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:54.922746    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np" podUID="e77e610e-005e-4e06-a301-161d60b6df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 29 22:31:55.217349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:55.217053    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:31:55.217349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:55.217118    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w" podUID="1bb026b1-e8ce-461a-bcdc-a38955de1842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 29 22:31:55.923216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:55.923181    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:31:55.923407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:55.923238    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np" podUID="e77e610e-005e-4e06-a301-161d60b6df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 29 22:31:56.212731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:56.212695    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:31:56.213293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:56.213263    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w" podUID="1bb026b1-e8ce-461a-bcdc-a38955de1842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 29 22:31:56.816838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:56.816803    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:56.926354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:56.922759    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:31:56.926354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:56.922823    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np" podUID="e77e610e-005e-4e06-a301-161d60b6df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 29 22:31:57.217549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.217520    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:31:57.217985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.217955    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w" podUID="1bb026b1-e8ce-461a-bcdc-a38955de1842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 29 22:31:57.547932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.547854    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"]
Oct 29 22:31:57.553520 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.553501    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:31:57.579355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.577268    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"]
Oct 29 22:31:57.599990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.585042    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:31:57.599990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.585756    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:31:57.642198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.642163    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n2qz\" (UniqueName: \"kubernetes.io/projected/2230d571-c982-421c-8515-fac6021c718f-kube-api-access-9n2qz\") pod \"maas-api-revoke-sub-fb2ac814-nfhp4\" (UID: \"2230d571-c982-421c-8515-fac6021c718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:31:57.642453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.642429    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2230d571-c982-421c-8515-fac6021c718f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb2ac814-nfhp4\" (UID: \"2230d571-c982-421c-8515-fac6021c718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:31:57.642493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.642470    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2230d571-c982-421c-8515-fac6021c718f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb2ac814-nfhp4\" (UID: \"2230d571-c982-421c-8515-fac6021c718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:31:57.649547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.649524    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:31:57.649649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.649569    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:31:57.744160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.744125    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2230d571-c982-421c-8515-fac6021c718f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb2ac814-nfhp4\" (UID: \"2230d571-c982-421c-8515-fac6021c718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:31:57.744160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.744168    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2230d571-c982-421c-8515-fac6021c718f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb2ac814-nfhp4\" (UID: \"2230d571-c982-421c-8515-fac6021c718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:31:57.744363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.744248    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2qz\" (UniqueName: \"kubernetes.io/projected/2230d571-c982-421c-8515-fac6021c718f-kube-api-access-9n2qz\") pod \"maas-api-revoke-sub-fb2ac814-nfhp4\" (UID: \"2230d571-c982-421c-8515-fac6021c718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:31:57.744895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.744872    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2230d571-c982-421c-8515-fac6021c718f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb2ac814-nfhp4\" (UID: \"2230d571-c982-421c-8515-fac6021c718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:31:57.746847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.746824    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2230d571-c982-421c-8515-fac6021c718f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb2ac814-nfhp4\" (UID: \"2230d571-c982-421c-8515-fac6021c718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:31:57.765174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.765144    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n2qz\" (UniqueName: \"kubernetes.io/projected/2230d571-c982-421c-8515-fac6021c718f-kube-api-access-9n2qz\") pod \"maas-api-revoke-sub-fb2ac814-nfhp4\" (UID: \"2230d571-c982-421c-8515-fac6021c718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:31:57.868383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.868302    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:31:57.924205 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.922845    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 29 22:31:57.924205 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:57.922899    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np" podUID="e77e610e-005e-4e06-a301-161d60b6df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 29 22:31:58.172337 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:31:58.172222    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2230d571_c982_421c_8515_fac6021c718f.slice/crio-fb1832c7ea0298255ffe7e97b75594fc5ce83bf498b35fa44a82775296ebeb6a WatchSource:0}: Error finding container fb1832c7ea0298255ffe7e97b75594fc5ce83bf498b35fa44a82775296ebeb6a: Status 404 returned error can't find the container with id fb1832c7ea0298255ffe7e97b75594fc5ce83bf498b35fa44a82775296ebeb6a
Oct 29 22:31:58.178548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.178524    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"]
Oct 29 22:31:58.209397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.209317    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:31:58.209554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.209420    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w" podUID="1bb026b1-e8ce-461a-bcdc-a38955de1842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 29 22:31:58.266951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.266918    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"]
Oct 29 22:31:58.272438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.272416    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:31:58.288606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.288581    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"]
Oct 29 22:31:58.352875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.352843    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmbhh\" (UniqueName: \"kubernetes.io/projected/14ffa1a1-4cb6-490a-a913-ad7c27348866-kube-api-access-qmbhh\") pod \"maas-api-revoke-sub-a458a371-xw9ch\" (UID: \"14ffa1a1-4cb6-490a-a913-ad7c27348866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:31:58.352992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.352939    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a458a371-xw9ch\" (UID: \"14ffa1a1-4cb6-490a-a913-ad7c27348866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:31:58.353046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.353033    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a458a371-xw9ch\" (UID: \"14ffa1a1-4cb6-490a-a913-ad7c27348866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:31:58.354822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.354771    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:58.443421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.443375    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4" event={"ID":"2230d571-c982-421c-8515-fac6021c718f","Type":"ContainerStarted","Data":"fb1832c7ea0298255ffe7e97b75594fc5ce83bf498b35fa44a82775296ebeb6a"}
Oct 29 22:31:58.454603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.454571    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a458a371-xw9ch\" (UID: \"14ffa1a1-4cb6-490a-a913-ad7c27348866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:31:58.454754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.454696    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmbhh\" (UniqueName: \"kubernetes.io/projected/14ffa1a1-4cb6-490a-a913-ad7c27348866-kube-api-access-qmbhh\") pod \"maas-api-revoke-sub-a458a371-xw9ch\" (UID: \"14ffa1a1-4cb6-490a-a913-ad7c27348866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:31:58.455168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.455146    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a458a371-xw9ch\" (UID: \"14ffa1a1-4cb6-490a-a913-ad7c27348866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:31:58.455473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.455447    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a458a371-xw9ch\" (UID: \"14ffa1a1-4cb6-490a-a913-ad7c27348866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:31:58.458474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.458448    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a458a371-xw9ch\" (UID: \"14ffa1a1-4cb6-490a-a913-ad7c27348866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:31:58.479398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.478652    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmbhh\" (UniqueName: \"kubernetes.io/projected/14ffa1a1-4cb6-490a-a913-ad7c27348866-kube-api-access-qmbhh\") pod \"maas-api-revoke-sub-a458a371-xw9ch\" (UID: \"14ffa1a1-4cb6-490a-a913-ad7c27348866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:31:58.589256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.589225    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:31:58.699290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.699185    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:58.815253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.815204    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:31:58.911902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:58.911863    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"]
Oct 29 22:31:59.145838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.143900    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:59.145838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.145096    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:59.147826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.147800    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:31:59.362336 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.362291    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:59.362938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.362911    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:59.364251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.364211    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:31:59.434514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.434486    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"]
Oct 29 22:31:59.439044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.439026    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:31:59.460229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.460158    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"]
Oct 29 22:31:59.476194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.476163    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch" event={"ID":"14ffa1a1-4cb6-490a-a913-ad7c27348866","Type":"ContainerStarted","Data":"4814adf441cb06d860e3538985f1d8969a8ab7229f29e5a0f0ff52c2546a9779"}
Oct 29 22:31:59.476344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.476211    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch" event={"ID":"14ffa1a1-4cb6-490a-a913-ad7c27348866","Type":"ContainerStarted","Data":"874b884bdafa7e2c9bf468802f1d7f6da8c13791ae70778b8d36dcdf65e90fd5"}
Oct 29 22:31:59.479504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.478717    4860 generic.go:356] "Generic (PLEG): container finished" podID="2230d571-c982-421c-8515-fac6021c718f" containerID="5fb3e05a0e073d06611cc0ca02fd64e95a6299d54dee794d971eeff6fe094b93" exitCode=0
Oct 29 22:31:59.479504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.478844    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4" event={"ID":"2230d571-c982-421c-8515-fac6021c718f","Type":"ContainerDied","Data":"5fb3e05a0e073d06611cc0ca02fd64e95a6299d54dee794d971eeff6fe094b93"}
Oct 29 22:31:59.554487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.554120    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch" podStartSLOduration=1.554104468 podStartE2EDuration="1.554104468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:59.549939449 +0000 UTC m=+1061.574715580" watchObservedRunningTime="2026-10-29 22:31:59.554104468 +0000 UTC m=+1061.578880643"
Oct 29 22:31:59.596600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.596563    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b703e63f-9cee-4556-be46-450232faaa43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cb742ee-tzl2m\" (UID: \"b703e63f-9cee-4556-be46-450232faaa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:31:59.596777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.596621    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvxjl\" (UniqueName: \"kubernetes.io/projected/b703e63f-9cee-4556-be46-450232faaa43-kube-api-access-rvxjl\") pod \"maas-api-revoke-sub-0cb742ee-tzl2m\" (UID: \"b703e63f-9cee-4556-be46-450232faaa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:31:59.597118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.596922    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b703e63f-9cee-4556-be46-450232faaa43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cb742ee-tzl2m\" (UID: \"b703e63f-9cee-4556-be46-450232faaa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:31:59.698396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.698362    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b703e63f-9cee-4556-be46-450232faaa43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cb742ee-tzl2m\" (UID: \"b703e63f-9cee-4556-be46-450232faaa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:31:59.698549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.698403    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvxjl\" (UniqueName: \"kubernetes.io/projected/b703e63f-9cee-4556-be46-450232faaa43-kube-api-access-rvxjl\") pod \"maas-api-revoke-sub-0cb742ee-tzl2m\" (UID: \"b703e63f-9cee-4556-be46-450232faaa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:31:59.698549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.698490    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b703e63f-9cee-4556-be46-450232faaa43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cb742ee-tzl2m\" (UID: \"b703e63f-9cee-4556-be46-450232faaa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:31:59.699082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.699056    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b703e63f-9cee-4556-be46-450232faaa43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cb742ee-tzl2m\" (UID: \"b703e63f-9cee-4556-be46-450232faaa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:31:59.700604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.700581    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b703e63f-9cee-4556-be46-450232faaa43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cb742ee-tzl2m\" (UID: \"b703e63f-9cee-4556-be46-450232faaa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:31:59.720274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.720226    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvxjl\" (UniqueName: \"kubernetes.io/projected/b703e63f-9cee-4556-be46-450232faaa43-kube-api-access-rvxjl\") pod \"maas-api-revoke-sub-0cb742ee-tzl2m\" (UID: \"b703e63f-9cee-4556-be46-450232faaa43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:31:59.758034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:31:59.758013    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:32:00.065792 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:00.065763    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb703e63f_9cee_4556_be46_450232faaa43.slice/crio-854a990adebfbd0ed8aafe0cf05dcb6a6603b91437f7ee055d998573e79e6e48 WatchSource:0}: Error finding container 854a990adebfbd0ed8aafe0cf05dcb6a6603b91437f7ee055d998573e79e6e48: Status 404 returned error can't find the container with id 854a990adebfbd0ed8aafe0cf05dcb6a6603b91437f7ee055d998573e79e6e48
Oct 29 22:32:00.072795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.072760    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"]
Oct 29 22:32:00.133821 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.133791    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:00.489538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.489503    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m" event={"ID":"b703e63f-9cee-4556-be46-450232faaa43","Type":"ContainerStarted","Data":"a3cd4ca44e1fd6f1d334a9a4ce03de76ee9b09fc9e2081de27a392c94521f31e"}
Oct 29 22:32:00.489916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.489565    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m" event={"ID":"b703e63f-9cee-4556-be46-450232faaa43","Type":"ContainerStarted","Data":"854a990adebfbd0ed8aafe0cf05dcb6a6603b91437f7ee055d998573e79e6e48"}
Oct 29 22:32:00.491175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.491145    4860 generic.go:356] "Generic (PLEG): container finished" podID="14ffa1a1-4cb6-490a-a913-ad7c27348866" containerID="4814adf441cb06d860e3538985f1d8969a8ab7229f29e5a0f0ff52c2546a9779" exitCode=0
Oct 29 22:32:00.491279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.491229    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch" event={"ID":"14ffa1a1-4cb6-490a-a913-ad7c27348866","Type":"ContainerDied","Data":"4814adf441cb06d860e3538985f1d8969a8ab7229f29e5a0f0ff52c2546a9779"}
Oct 29 22:32:00.533565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.533511    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m" podStartSLOduration=1.533496832 podStartE2EDuration="1.533496832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:00.529437663 +0000 UTC m=+1062.554213796" watchObservedRunningTime="2026-10-29 22:32:00.533496832 +0000 UTC m=+1062.558272962"
Oct 29 22:32:00.632095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.632067    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:32:00.814964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.814926    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2230d571-c982-421c-8515-fac6021c718f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2230d571-c982-421c-8515-fac6021c718f-maas-api-openshift-ca\") pod \"2230d571-c982-421c-8515-fac6021c718f\" (UID: \"2230d571-c982-421c-8515-fac6021c718f\") "
Oct 29 22:32:00.815111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.815071    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2230d571-c982-421c-8515-fac6021c718f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2230d571-c982-421c-8515-fac6021c718f-maas-api-xks-ca\") pod \"2230d571-c982-421c-8515-fac6021c718f\" (UID: \"2230d571-c982-421c-8515-fac6021c718f\") "
Oct 29 22:32:00.815164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.815116    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2230d571-c982-421c-8515-fac6021c718f-kube-api-access-9n2qz\" (UniqueName: \"kubernetes.io/projected/2230d571-c982-421c-8515-fac6021c718f-kube-api-access-9n2qz\") pod \"2230d571-c982-421c-8515-fac6021c718f\" (UID: \"2230d571-c982-421c-8515-fac6021c718f\") "
Oct 29 22:32:00.815400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.815375    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2230d571-c982-421c-8515-fac6021c718f-maas-api-openshift-ca" pod "2230d571-c982-421c-8515-fac6021c718f" (UID: "2230d571-c982-421c-8515-fac6021c718f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:00.815764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.815739    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2230d571-c982-421c-8515-fac6021c718f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:00.817189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.817166    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2230d571-c982-421c-8515-fac6021c718f-maas-api-xks-ca" pod "2230d571-c982-421c-8515-fac6021c718f" (UID: "2230d571-c982-421c-8515-fac6021c718f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:00.817277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.817252    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2230d571-c982-421c-8515-fac6021c718f-kube-api-access-9n2qz" pod "2230d571-c982-421c-8515-fac6021c718f" (UID: "2230d571-c982-421c-8515-fac6021c718f"). InnerVolumeSpecName "kube-api-access-9n2qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:00.917243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.917211    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2230d571-c982-421c-8515-fac6021c718f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:00.917243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:00.917239    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n2qz\" (UniqueName: \"kubernetes.io/projected/2230d571-c982-421c-8515-fac6021c718f-kube-api-access-9n2qz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:01.500814 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.500782    4860 generic.go:356] "Generic (PLEG): container finished" podID="b703e63f-9cee-4556-be46-450232faaa43" containerID="a3cd4ca44e1fd6f1d334a9a4ce03de76ee9b09fc9e2081de27a392c94521f31e" exitCode=0
Oct 29 22:32:01.501212 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.500868    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m" event={"ID":"b703e63f-9cee-4556-be46-450232faaa43","Type":"ContainerDied","Data":"a3cd4ca44e1fd6f1d334a9a4ce03de76ee9b09fc9e2081de27a392c94521f31e"}
Oct 29 22:32:01.502441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.502426    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"
Oct 29 22:32:01.502526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.502444    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4" event={"ID":"2230d571-c982-421c-8515-fac6021c718f","Type":"ContainerDied","Data":"fb1832c7ea0298255ffe7e97b75594fc5ce83bf498b35fa44a82775296ebeb6a"}
Oct 29 22:32:01.502526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.502482    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb1832c7ea0298255ffe7e97b75594fc5ce83bf498b35fa44a82775296ebeb6a"
Oct 29 22:32:01.672794 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.672770    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:32:01.720269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.720214    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:01.835585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.835491    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-openshift-ca\") pod \"14ffa1a1-4cb6-490a-a913-ad7c27348866\" (UID: \"14ffa1a1-4cb6-490a-a913-ad7c27348866\") "
Oct 29 22:32:01.835757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.835589    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-xks-ca\") pod \"14ffa1a1-4cb6-490a-a913-ad7c27348866\" (UID: \"14ffa1a1-4cb6-490a-a913-ad7c27348866\") "
Oct 29 22:32:01.835813 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.835798    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ffa1a1-4cb6-490a-a913-ad7c27348866-kube-api-access-qmbhh\" (UniqueName: \"kubernetes.io/projected/14ffa1a1-4cb6-490a-a913-ad7c27348866-kube-api-access-qmbhh\") pod \"14ffa1a1-4cb6-490a-a913-ad7c27348866\" (UID: \"14ffa1a1-4cb6-490a-a913-ad7c27348866\") "
Oct 29 22:32:01.835895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.835868    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-openshift-ca" pod "14ffa1a1-4cb6-490a-a913-ad7c27348866" (UID: "14ffa1a1-4cb6-490a-a913-ad7c27348866"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:01.836269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.836253    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:01.838072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.838024    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-xks-ca" pod "14ffa1a1-4cb6-490a-a913-ad7c27348866" (UID: "14ffa1a1-4cb6-490a-a913-ad7c27348866"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:01.838282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.838242    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ffa1a1-4cb6-490a-a913-ad7c27348866-kube-api-access-qmbhh" pod "14ffa1a1-4cb6-490a-a913-ad7c27348866" (UID: "14ffa1a1-4cb6-490a-a913-ad7c27348866"). InnerVolumeSpecName "kube-api-access-qmbhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:01.937229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.937195    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ffa1a1-4cb6-490a-a913-ad7c27348866-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:01.937229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:01.937223    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmbhh\" (UniqueName: \"kubernetes.io/projected/14ffa1a1-4cb6-490a-a913-ad7c27348866-kube-api-access-qmbhh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:02.512893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.512858    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"
Oct 29 22:32:02.512893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.512880    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch" event={"ID":"14ffa1a1-4cb6-490a-a913-ad7c27348866","Type":"ContainerDied","Data":"874b884bdafa7e2c9bf468802f1d7f6da8c13791ae70778b8d36dcdf65e90fd5"}
Oct 29 22:32:02.513350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.512922    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="874b884bdafa7e2c9bf468802f1d7f6da8c13791ae70778b8d36dcdf65e90fd5"
Oct 29 22:32:02.682164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.682138    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:32:02.744808 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.744778    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b703e63f-9cee-4556-be46-450232faaa43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b703e63f-9cee-4556-be46-450232faaa43-maas-api-xks-ca\") pod \"b703e63f-9cee-4556-be46-450232faaa43\" (UID: \"b703e63f-9cee-4556-be46-450232faaa43\") "
Oct 29 22:32:02.744992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.744973    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b703e63f-9cee-4556-be46-450232faaa43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b703e63f-9cee-4556-be46-450232faaa43-maas-api-openshift-ca\") pod \"b703e63f-9cee-4556-be46-450232faaa43\" (UID: \"b703e63f-9cee-4556-be46-450232faaa43\") "
Oct 29 22:32:02.745143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.745127    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b703e63f-9cee-4556-be46-450232faaa43-kube-api-access-rvxjl\" (UniqueName: \"kubernetes.io/projected/b703e63f-9cee-4556-be46-450232faaa43-kube-api-access-rvxjl\") pod \"b703e63f-9cee-4556-be46-450232faaa43\" (UID: \"b703e63f-9cee-4556-be46-450232faaa43\") "
Oct 29 22:32:02.745482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.745441    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b703e63f-9cee-4556-be46-450232faaa43-maas-api-openshift-ca" pod "b703e63f-9cee-4556-be46-450232faaa43" (UID: "b703e63f-9cee-4556-be46-450232faaa43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:02.745747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.745724    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b703e63f-9cee-4556-be46-450232faaa43-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:02.747531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.747502    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b703e63f-9cee-4556-be46-450232faaa43-maas-api-xks-ca" pod "b703e63f-9cee-4556-be46-450232faaa43" (UID: "b703e63f-9cee-4556-be46-450232faaa43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:02.747603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.747562    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b703e63f-9cee-4556-be46-450232faaa43-kube-api-access-rvxjl" pod "b703e63f-9cee-4556-be46-450232faaa43" (UID: "b703e63f-9cee-4556-be46-450232faaa43"). InnerVolumeSpecName "kube-api-access-rvxjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:02.847554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.847489    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvxjl\" (UniqueName: \"kubernetes.io/projected/b703e63f-9cee-4556-be46-450232faaa43-kube-api-access-rvxjl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:02.847554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:02.847516    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b703e63f-9cee-4556-be46-450232faaa43-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:03.371617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:03.371561    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:03.523063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:03.523035    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"
Oct 29 22:32:03.523063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:03.523062    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m" event={"ID":"b703e63f-9cee-4556-be46-450232faaa43","Type":"ContainerDied","Data":"854a990adebfbd0ed8aafe0cf05dcb6a6603b91437f7ee055d998573e79e6e48"}
Oct 29 22:32:03.523553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:03.523105    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="854a990adebfbd0ed8aafe0cf05dcb6a6603b91437f7ee055d998573e79e6e48"
Oct 29 22:32:04.290463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.290432    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"]
Oct 29 22:32:04.291048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.291032    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2230d571-c982-421c-8515-fac6021c718f" containerName="revoke-keys"
Oct 29 22:32:04.291048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.291047    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2230d571-c982-421c-8515-fac6021c718f" containerName="revoke-keys"
Oct 29 22:32:04.291202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.291070    4860 cpu_manager.go:414] "containerMap: removing container" podUID="14ffa1a1-4cb6-490a-a913-ad7c27348866" containerName="revoke-keys"
Oct 29 22:32:04.291202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.291075    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ffa1a1-4cb6-490a-a913-ad7c27348866" containerName="revoke-keys"
Oct 29 22:32:04.291202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.291082    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b703e63f-9cee-4556-be46-450232faaa43" containerName="revoke-keys"
Oct 29 22:32:04.291202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.291088    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b703e63f-9cee-4556-be46-450232faaa43" containerName="revoke-keys"
Oct 29 22:32:04.291202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.291203    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2230d571-c982-421c-8515-fac6021c718f" containerName="revoke-keys"
Oct 29 22:32:04.291555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.291215    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ffa1a1-4cb6-490a-a913-ad7c27348866" containerName="revoke-keys"
Oct 29 22:32:04.291555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.291220    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b703e63f-9cee-4556-be46-450232faaa43" containerName="revoke-keys"
Oct 29 22:32:04.295227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.295205    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:04.312756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.312729    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"]
Oct 29 22:32:04.319727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.319702    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:32:04.319842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.319809    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:32:04.319842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.319827    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:32:04.325138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.325118    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:32:04.368982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.368958    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f22ac5-hdjdb\" (UID: \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:04.369102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.369089    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-848vd\" (UniqueName: \"kubernetes.io/projected/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-kube-api-access-848vd\") pod \"maas-api-revoke-sub-89f22ac5-hdjdb\" (UID: \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:04.369143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.369125    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f22ac5-hdjdb\" (UID: \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:04.470400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.470364    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f22ac5-hdjdb\" (UID: \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:04.470589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.470495    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-848vd\" (UniqueName: \"kubernetes.io/projected/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-kube-api-access-848vd\") pod \"maas-api-revoke-sub-89f22ac5-hdjdb\" (UID: \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:04.470589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.470541    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f22ac5-hdjdb\" (UID: \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:04.471554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.471524    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f22ac5-hdjdb\" (UID: \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:04.473406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.473379    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f22ac5-hdjdb\" (UID: \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:04.491632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.491610    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-848vd\" (UniqueName: \"kubernetes.io/projected/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-kube-api-access-848vd\") pod \"maas-api-revoke-sub-89f22ac5-hdjdb\" (UID: \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:04.613305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.613236    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:04.917001 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:04.916967    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9175cc_da29_4c15_a2c3_a32b7196fd0c.slice/crio-32156ddb59830ab2887bc6f5f8c7c0da370db053d7f952f59e85373eaec4bd31 WatchSource:0}: Error finding container 32156ddb59830ab2887bc6f5f8c7c0da370db053d7f952f59e85373eaec4bd31: Status 404 returned error can't find the container with id 32156ddb59830ab2887bc6f5f8c7c0da370db053d7f952f59e85373eaec4bd31
Oct 29 22:32:04.920543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:04.920519    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"]
Oct 29 22:32:05.120529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:05.120504    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:05.543267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:05.543231    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb" event={"ID":"5a9175cc-da29-4c15-a2c3-a32b7196fd0c","Type":"ContainerStarted","Data":"f5821400627e979e250cfb51572266749dcc1f08faefdc06f1bef07ffdfd3e28"}
Oct 29 22:32:05.543468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:05.543284    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb" event={"ID":"5a9175cc-da29-4c15-a2c3-a32b7196fd0c","Type":"ContainerStarted","Data":"32156ddb59830ab2887bc6f5f8c7c0da370db053d7f952f59e85373eaec4bd31"}
Oct 29 22:32:05.584545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:05.584494    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb" podStartSLOduration=1.584479779 podStartE2EDuration="1.584479779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:05.582261396 +0000 UTC m=+1067.607037526" watchObservedRunningTime="2026-10-29 22:32:05.584479779 +0000 UTC m=+1067.609255910"
Oct 29 22:32:06.553186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:06.553154    4860 generic.go:356] "Generic (PLEG): container finished" podID="5a9175cc-da29-4c15-a2c3-a32b7196fd0c" containerID="f5821400627e979e250cfb51572266749dcc1f08faefdc06f1bef07ffdfd3e28" exitCode=0
Oct 29 22:32:06.553634 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:06.553241    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb" event={"ID":"5a9175cc-da29-4c15-a2c3-a32b7196fd0c","Type":"ContainerDied","Data":"f5821400627e979e250cfb51572266749dcc1f08faefdc06f1bef07ffdfd3e28"}
Oct 29 22:32:06.705202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:06.705171    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:07.712171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:07.712142    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:07.812538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:07.812472    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-openshift-ca\") pod \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\" (UID: \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\") "
Oct 29 22:32:07.812538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:07.812506    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-xks-ca\") pod \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\" (UID: \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\") "
Oct 29 22:32:07.812698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:07.812579    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-kube-api-access-848vd\" (UniqueName: \"kubernetes.io/projected/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-kube-api-access-848vd\") pod \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\" (UID: \"5a9175cc-da29-4c15-a2c3-a32b7196fd0c\") "
Oct 29 22:32:07.812805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:07.812784    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-openshift-ca" pod "5a9175cc-da29-4c15-a2c3-a32b7196fd0c" (UID: "5a9175cc-da29-4c15-a2c3-a32b7196fd0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:07.813056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:07.813040    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:07.814683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:07.814658    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-kube-api-access-848vd" pod "5a9175cc-da29-4c15-a2c3-a32b7196fd0c" (UID: "5a9175cc-da29-4c15-a2c3-a32b7196fd0c"). InnerVolumeSpecName "kube-api-access-848vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:07.814773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:07.814694    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-xks-ca" pod "5a9175cc-da29-4c15-a2c3-a32b7196fd0c" (UID: "5a9175cc-da29-4c15-a2c3-a32b7196fd0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:07.930347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:07.914617    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:07.930614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:07.930590    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-848vd\" (UniqueName: \"kubernetes.io/projected/5a9175cc-da29-4c15-a2c3-a32b7196fd0c-kube-api-access-848vd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:08.273467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:08.273427    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:08.610002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:08.609963    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb" event={"ID":"5a9175cc-da29-4c15-a2c3-a32b7196fd0c","Type":"ContainerDied","Data":"32156ddb59830ab2887bc6f5f8c7c0da370db053d7f952f59e85373eaec4bd31"}
Oct 29 22:32:08.610259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:08.610244    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32156ddb59830ab2887bc6f5f8c7c0da370db053d7f952f59e85373eaec4bd31"
Oct 29 22:32:08.610430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:08.610417    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"
Oct 29 22:32:09.827825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:09.827774    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:32:09.837430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:09.837394    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:09.896362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:09.886422    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:09.979467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:09.979421    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.020477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.020427    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.050663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.050616    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.076346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.074423    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.195442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.195403    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:32:10.221454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.221401    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.296888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.296838    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.339472 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.339421    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.340862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.340830    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.883363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.838684    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.883363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.841598    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:32:10.883363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.876539    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:32:10.883363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.876719    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:32:10.909594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.909498    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:10.969845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:10.912316    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:32:11.216873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.146900    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:11.216873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.147394    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:32:11.216873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.148918    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:32:11.257350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.219266    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:32:11.286368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.259851    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:11.480606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.480505    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:11.482728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.481055    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:32:11.482893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.481474    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:32:11.483033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.481499    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:11.483134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.481530    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:32:11.483228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.482033    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:32:11.532765 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.532722    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:11.562380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.537801    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:32:11.562380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.544017    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:32:11.562380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.544057    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:32:11.562380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.544093    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:32:11.562380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.543879    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 29 22:32:11.565267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.565243    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:32:11.565559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.565543    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:32:11.891406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.850987    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.313372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.923388    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.313372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:11.923429    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.313372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:12.151836    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.313372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:12.151891    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.537131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:12.510770    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:12.568401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:12.560780    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.690276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:12.568417    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.768108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:12.568475    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.775145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:12.568533    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:12.791672 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:12.568554    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:32:12.791980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:12.743400    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.225350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.205589    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded"
Oct 29 22:32:13.299347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.298481    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:32:13.299551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.298413    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Oct 29 22:32:13.307345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.303388    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.307345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.303673    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.340973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.339876    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.524505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.524408    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.529348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.525804    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.620377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.618413    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.630348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.629463    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:32:13.630348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.629527    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" podUID="fb326055-bb08-42f0-8597-ddfe942b2ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.781399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.780807    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:32:13.781399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.781186    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:32:13.789346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.783042    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.915554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.915510    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:13.934169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:13.920172    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:14.121905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.121790    4860 patch_prober.go:29] interesting pod/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:32:14.122174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.122141    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s" podUID="9bb10043-0048-4595-bc98-b3333f888d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:14.308091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.308048    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:14.311488 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.311455    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:32:14.315920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.315887    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:14.316389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.316366    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:32:14.348366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.340204    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:14.467525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.467477    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:32:14.517359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.512383    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:32:14.697363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.538442    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:32:14.697363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.664965    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:14.750387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.727010    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:32:14.781568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.781530    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:14.784770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.784728    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:32:14.841858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.841823    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:32:14.851153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.851122    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"]
Oct 29 22:32:14.852113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.852087    4860 cpu_manager.go:414] "containerMap: removing container" podUID="5a9175cc-da29-4c15-a2c3-a32b7196fd0c" containerName="revoke-keys"
Oct 29 22:32:14.852113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.852113    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9175cc-da29-4c15-a2c3-a32b7196fd0c" containerName="revoke-keys"
Oct 29 22:32:14.860724 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.860691    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9175cc-da29-4c15-a2c3-a32b7196fd0c" containerName="revoke-keys"
Oct 29 22:32:14.878524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.877582    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:32:14.878524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.877622    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"]
Oct 29 22:32:14.878524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.878471    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:32:14.878524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.878514    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:32:14.878834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.878545    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:32:14.878834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.878576    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:32:14.878834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.878606    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:32:14.878834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.878819    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:14.880970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.880925    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:32:14.901343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.898824    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:32:14.901343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.898865    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:32:14.901343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.898977    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:32:14.901343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.899029    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:32:14.983189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.983159    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-730de389-9vpvh\" (UID: \"8f370020-4408-4381-9edd-9ecfd8228ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:14.983484 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.983446    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-730de389-9vpvh\" (UID: \"8f370020-4408-4381-9edd-9ecfd8228ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:14.983849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:14.983823    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv2vh\" (UniqueName: \"kubernetes.io/projected/8f370020-4408-4381-9edd-9ecfd8228ad8-kube-api-access-hv2vh\") pod \"maas-api-revoke-sub-730de389-9vpvh\" (UID: \"8f370020-4408-4381-9edd-9ecfd8228ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:15.085648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.085561    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv2vh\" (UniqueName: \"kubernetes.io/projected/8f370020-4408-4381-9edd-9ecfd8228ad8-kube-api-access-hv2vh\") pod \"maas-api-revoke-sub-730de389-9vpvh\" (UID: \"8f370020-4408-4381-9edd-9ecfd8228ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:15.085648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.085631    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-730de389-9vpvh\" (UID: \"8f370020-4408-4381-9edd-9ecfd8228ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:15.085846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.085753    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-730de389-9vpvh\" (UID: \"8f370020-4408-4381-9edd-9ecfd8228ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:15.086260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.086236    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-730de389-9vpvh\" (UID: \"8f370020-4408-4381-9edd-9ecfd8228ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:15.088161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.088137    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-730de389-9vpvh\" (UID: \"8f370020-4408-4381-9edd-9ecfd8228ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:15.111127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.111098    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv2vh\" (UniqueName: \"kubernetes.io/projected/8f370020-4408-4381-9edd-9ecfd8228ad8-kube-api-access-hv2vh\") pod \"maas-api-revoke-sub-730de389-9vpvh\" (UID: \"8f370020-4408-4381-9edd-9ecfd8228ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:15.213628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.213599    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:15.229621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.229592    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"]
Oct 29 22:32:15.234138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.234119    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:15.277216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.277179    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"]
Oct 29 22:32:15.290140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.290111    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1d54390-27g4s\" (UID: \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:15.290563 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.290307    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfmdn\" (UniqueName: \"kubernetes.io/projected/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-kube-api-access-pfmdn\") pod \"maas-api-revoke-sub-b1d54390-27g4s\" (UID: \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:15.290563 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.290407    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1d54390-27g4s\" (UID: \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:15.391957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.391875    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1d54390-27g4s\" (UID: \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:15.392389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.391998    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfmdn\" (UniqueName: \"kubernetes.io/projected/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-kube-api-access-pfmdn\") pod \"maas-api-revoke-sub-b1d54390-27g4s\" (UID: \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:15.392389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.392366    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1d54390-27g4s\" (UID: \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:15.392936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.392913    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1d54390-27g4s\" (UID: \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:15.394392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.394369    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1d54390-27g4s\" (UID: \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:15.412118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.412097    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfmdn\" (UniqueName: \"kubernetes.io/projected/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-kube-api-access-pfmdn\") pod \"maas-api-revoke-sub-b1d54390-27g4s\" (UID: \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:15.521868 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:15.521828    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f370020_4408_4381_9edd_9ecfd8228ad8.slice/crio-a5099ce12a200d1a2a59be503a0f108ff9b7220a05a4d90f46e8271c444bb04a WatchSource:0}: Error finding container a5099ce12a200d1a2a59be503a0f108ff9b7220a05a4d90f46e8271c444bb04a: Status 404 returned error can't find the container with id a5099ce12a200d1a2a59be503a0f108ff9b7220a05a4d90f46e8271c444bb04a
Oct 29 22:32:15.526524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.526501    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"]
Oct 29 22:32:15.568900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.568865    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:15.694636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.694605    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"]
Oct 29 22:32:15.696461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.696434    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:15.700579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.700563    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:15.721367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.720355    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"]
Oct 29 22:32:15.798305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.798276    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwzw\" (UniqueName: \"kubernetes.io/projected/7fe55b30-3f30-4718-99e3-7144caa7da9b-kube-api-access-qtwzw\") pod \"maas-api-revoke-sub-6dcfad3b-hkt4l\" (UID: \"7fe55b30-3f30-4718-99e3-7144caa7da9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:15.798462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.798353    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dcfad3b-hkt4l\" (UID: \"7fe55b30-3f30-4718-99e3-7144caa7da9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:15.798616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.798598    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dcfad3b-hkt4l\" (UID: \"7fe55b30-3f30-4718-99e3-7144caa7da9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:15.875388 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:15.875355    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a71204_5e6e_4ee4_aae2_6daeabe95b7c.slice/crio-d2bccfcd5eb121e2241af264f5e263900927d52b72cac8aad10b7d1a1e463a78 WatchSource:0}: Error finding container d2bccfcd5eb121e2241af264f5e263900927d52b72cac8aad10b7d1a1e463a78: Status 404 returned error can't find the container with id d2bccfcd5eb121e2241af264f5e263900927d52b72cac8aad10b7d1a1e463a78
Oct 29 22:32:15.890097 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.883704    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"]
Oct 29 22:32:15.900504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.900471    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dcfad3b-hkt4l\" (UID: \"7fe55b30-3f30-4718-99e3-7144caa7da9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:15.901022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.900993    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwzw\" (UniqueName: \"kubernetes.io/projected/7fe55b30-3f30-4718-99e3-7144caa7da9b-kube-api-access-qtwzw\") pod \"maas-api-revoke-sub-6dcfad3b-hkt4l\" (UID: \"7fe55b30-3f30-4718-99e3-7144caa7da9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:15.901127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.901071    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dcfad3b-hkt4l\" (UID: \"7fe55b30-3f30-4718-99e3-7144caa7da9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:15.901960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.901940    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dcfad3b-hkt4l\" (UID: \"7fe55b30-3f30-4718-99e3-7144caa7da9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:15.905681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.905658    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dcfad3b-hkt4l\" (UID: \"7fe55b30-3f30-4718-99e3-7144caa7da9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:15.922898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:15.922875    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwzw\" (UniqueName: \"kubernetes.io/projected/7fe55b30-3f30-4718-99e3-7144caa7da9b-kube-api-access-qtwzw\") pod \"maas-api-revoke-sub-6dcfad3b-hkt4l\" (UID: \"7fe55b30-3f30-4718-99e3-7144caa7da9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:16.016109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:16.016038    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:16.324295 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:16.324265    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe55b30_3f30_4718_99e3_7144caa7da9b.slice/crio-ea0245359c05f2fbcf78bd9ce0500084e951f76cbaeeaf280be7d823921537c6 WatchSource:0}: Error finding container ea0245359c05f2fbcf78bd9ce0500084e951f76cbaeeaf280be7d823921537c6: Status 404 returned error can't find the container with id ea0245359c05f2fbcf78bd9ce0500084e951f76cbaeeaf280be7d823921537c6
Oct 29 22:32:16.328844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:16.328823    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"]
Oct 29 22:32:16.514792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:16.514562    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s" event={"ID":"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c","Type":"ContainerStarted","Data":"1d102096c87c5d0f4d3f29b882f82a8b19200fd73f5c5e6c521e798d11cb834b"}
Oct 29 22:32:16.514792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:16.514621    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s" event={"ID":"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c","Type":"ContainerStarted","Data":"d2bccfcd5eb121e2241af264f5e263900927d52b72cac8aad10b7d1a1e463a78"}
Oct 29 22:32:16.516491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:16.516458    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l" event={"ID":"7fe55b30-3f30-4718-99e3-7144caa7da9b","Type":"ContainerStarted","Data":"ea0245359c05f2fbcf78bd9ce0500084e951f76cbaeeaf280be7d823921537c6"}
Oct 29 22:32:16.518704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:16.518677    4860 generic.go:356] "Generic (PLEG): container finished" podID="8f370020-4408-4381-9edd-9ecfd8228ad8" containerID="70c5020fb95e96050b2248114548f7ed7be7c5e04f2325881bca31950751df66" exitCode=0
Oct 29 22:32:16.518829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:16.518790    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh" event={"ID":"8f370020-4408-4381-9edd-9ecfd8228ad8","Type":"ContainerDied","Data":"70c5020fb95e96050b2248114548f7ed7be7c5e04f2325881bca31950751df66"}
Oct 29 22:32:16.518903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:16.518829    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh" event={"ID":"8f370020-4408-4381-9edd-9ecfd8228ad8","Type":"ContainerStarted","Data":"a5099ce12a200d1a2a59be503a0f108ff9b7220a05a4d90f46e8271c444bb04a"}
Oct 29 22:32:16.560074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:16.559997    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s" podStartSLOduration=1.55998437 podStartE2EDuration="1.55998437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:16.558243104 +0000 UTC m=+1078.583019256" watchObservedRunningTime="2026-10-29 22:32:16.55998437 +0000 UTC m=+1078.584760498"
Oct 29 22:32:17.433529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.433499    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:17.533393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.533360    4860 generic.go:356] "Generic (PLEG): container finished" podID="7fe55b30-3f30-4718-99e3-7144caa7da9b" containerID="a5454b254abba345333daafe77013300deadd340da40a1dd54e288cf7243f24a" exitCode=0
Oct 29 22:32:17.533853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.533393    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l" event={"ID":"7fe55b30-3f30-4718-99e3-7144caa7da9b","Type":"ContainerDied","Data":"a5454b254abba345333daafe77013300deadd340da40a1dd54e288cf7243f24a"}
Oct 29 22:32:17.535833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.535806    4860 generic.go:356] "Generic (PLEG): container finished" podID="e2a71204-5e6e-4ee4-aae2-6daeabe95b7c" containerID="1d102096c87c5d0f4d3f29b882f82a8b19200fd73f5c5e6c521e798d11cb834b" exitCode=0
Oct 29 22:32:17.535980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.535890    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s" event={"ID":"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c","Type":"ContainerDied","Data":"1d102096c87c5d0f4d3f29b882f82a8b19200fd73f5c5e6c521e798d11cb834b"}
Oct 29 22:32:17.711514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.711491    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:17.726668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.726644    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-xks-ca\") pod \"8f370020-4408-4381-9edd-9ecfd8228ad8\" (UID: \"8f370020-4408-4381-9edd-9ecfd8228ad8\") "
Oct 29 22:32:17.726786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.726773    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f370020-4408-4381-9edd-9ecfd8228ad8-kube-api-access-hv2vh\" (UniqueName: \"kubernetes.io/projected/8f370020-4408-4381-9edd-9ecfd8228ad8-kube-api-access-hv2vh\") pod \"8f370020-4408-4381-9edd-9ecfd8228ad8\" (UID: \"8f370020-4408-4381-9edd-9ecfd8228ad8\") "
Oct 29 22:32:17.726921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.726906    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-openshift-ca\") pod \"8f370020-4408-4381-9edd-9ecfd8228ad8\" (UID: \"8f370020-4408-4381-9edd-9ecfd8228ad8\") "
Oct 29 22:32:17.727291 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.727260    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-openshift-ca" pod "8f370020-4408-4381-9edd-9ecfd8228ad8" (UID: "8f370020-4408-4381-9edd-9ecfd8228ad8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:17.727658 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.727637    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:17.728727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.728704    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-xks-ca" pod "8f370020-4408-4381-9edd-9ecfd8228ad8" (UID: "8f370020-4408-4381-9edd-9ecfd8228ad8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:17.728926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.728910    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f370020-4408-4381-9edd-9ecfd8228ad8-kube-api-access-hv2vh" pod "8f370020-4408-4381-9edd-9ecfd8228ad8" (UID: "8f370020-4408-4381-9edd-9ecfd8228ad8"). InnerVolumeSpecName "kube-api-access-hv2vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:17.829149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.829119    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f370020-4408-4381-9edd-9ecfd8228ad8-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:17.829149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:17.829144    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv2vh\" (UniqueName: \"kubernetes.io/projected/8f370020-4408-4381-9edd-9ecfd8228ad8-kube-api-access-hv2vh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:18.054999 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:32:18.054962    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:32:18.545903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.545873    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"
Oct 29 22:32:18.545903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.545889    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh" event={"ID":"8f370020-4408-4381-9edd-9ecfd8228ad8","Type":"ContainerDied","Data":"a5099ce12a200d1a2a59be503a0f108ff9b7220a05a4d90f46e8271c444bb04a"}
Oct 29 22:32:18.546335 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.545944    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5099ce12a200d1a2a59be503a0f108ff9b7220a05a4d90f46e8271c444bb04a"
Oct 29 22:32:18.730340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.730301    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:18.738733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.738706    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-openshift-ca\") pod \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\" (UID: \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\") "
Oct 29 22:32:18.738891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.738864    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-xks-ca\") pod \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\" (UID: \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\") "
Oct 29 22:32:18.739034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.739015    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-kube-api-access-pfmdn\" (UniqueName: \"kubernetes.io/projected/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-kube-api-access-pfmdn\") pod \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\" (UID: \"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c\") "
Oct 29 22:32:18.739189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.739167    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-openshift-ca" pod "e2a71204-5e6e-4ee4-aae2-6daeabe95b7c" (UID: "e2a71204-5e6e-4ee4-aae2-6daeabe95b7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:18.740234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.740042    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:18.741461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.741432    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-xks-ca" pod "e2a71204-5e6e-4ee4-aae2-6daeabe95b7c" (UID: "e2a71204-5e6e-4ee4-aae2-6daeabe95b7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:18.742311 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.742279    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-kube-api-access-pfmdn" pod "e2a71204-5e6e-4ee4-aae2-6daeabe95b7c" (UID: "e2a71204-5e6e-4ee4-aae2-6daeabe95b7c"). InnerVolumeSpecName "kube-api-access-pfmdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:18.768301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.768274    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:18.840559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.840468    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-openshift-ca\") pod \"7fe55b30-3f30-4718-99e3-7144caa7da9b\" (UID: \"7fe55b30-3f30-4718-99e3-7144caa7da9b\") "
Oct 29 22:32:18.840559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.840546    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe55b30-3f30-4718-99e3-7144caa7da9b-kube-api-access-qtwzw\" (UniqueName: \"kubernetes.io/projected/7fe55b30-3f30-4718-99e3-7144caa7da9b-kube-api-access-qtwzw\") pod \"7fe55b30-3f30-4718-99e3-7144caa7da9b\" (UID: \"7fe55b30-3f30-4718-99e3-7144caa7da9b\") "
Oct 29 22:32:18.840780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.840693    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-xks-ca\") pod \"7fe55b30-3f30-4718-99e3-7144caa7da9b\" (UID: \"7fe55b30-3f30-4718-99e3-7144caa7da9b\") "
Oct 29 22:32:18.840899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.840869    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-openshift-ca" pod "7fe55b30-3f30-4718-99e3-7144caa7da9b" (UID: "7fe55b30-3f30-4718-99e3-7144caa7da9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:18.841260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.841240    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:18.841381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.841262    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:18.841381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.841273    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfmdn\" (UniqueName: \"kubernetes.io/projected/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c-kube-api-access-pfmdn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:18.843295 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.843266    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe55b30-3f30-4718-99e3-7144caa7da9b-kube-api-access-qtwzw" pod "7fe55b30-3f30-4718-99e3-7144caa7da9b" (UID: "7fe55b30-3f30-4718-99e3-7144caa7da9b"). InnerVolumeSpecName "kube-api-access-qtwzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:18.843411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.843382    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-xks-ca" pod "7fe55b30-3f30-4718-99e3-7144caa7da9b" (UID: "7fe55b30-3f30-4718-99e3-7144caa7da9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:18.942967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.942938    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe55b30-3f30-4718-99e3-7144caa7da9b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:18.942967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.942968    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtwzw\" (UniqueName: \"kubernetes.io/projected/7fe55b30-3f30-4718-99e3-7144caa7da9b-kube-api-access-qtwzw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:18.959190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:18.959144    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:19.557244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:19.557215    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"
Oct 29 22:32:19.557783 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:19.557257    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s" event={"ID":"e2a71204-5e6e-4ee4-aae2-6daeabe95b7c","Type":"ContainerDied","Data":"d2bccfcd5eb121e2241af264f5e263900927d52b72cac8aad10b7d1a1e463a78"}
Oct 29 22:32:19.557783 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:19.557303    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2bccfcd5eb121e2241af264f5e263900927d52b72cac8aad10b7d1a1e463a78"
Oct 29 22:32:19.558916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:19.558895    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"
Oct 29 22:32:19.559049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:19.558937    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l" event={"ID":"7fe55b30-3f30-4718-99e3-7144caa7da9b","Type":"ContainerDied","Data":"ea0245359c05f2fbcf78bd9ce0500084e951f76cbaeeaf280be7d823921537c6"}
Oct 29 22:32:19.559049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:19.558977    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea0245359c05f2fbcf78bd9ce0500084e951f76cbaeeaf280be7d823921537c6"
Oct 29 22:32:20.462806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:20.462762    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:22.137424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:22.137393    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:22.345801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:22.345770    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:22.470701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:22.470673    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:23.650152 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:23.650122    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:25.230679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.230644    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:25.624703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.624631    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"]
Oct 29 22:32:25.625314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.625288    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e2a71204-5e6e-4ee4-aae2-6daeabe95b7c" containerName="revoke-keys"
Oct 29 22:32:25.625314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.625314    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a71204-5e6e-4ee4-aae2-6daeabe95b7c" containerName="revoke-keys"
Oct 29 22:32:25.625432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.625357    4860 cpu_manager.go:414] "containerMap: removing container" podUID="7fe55b30-3f30-4718-99e3-7144caa7da9b" containerName="revoke-keys"
Oct 29 22:32:25.625432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.625363    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe55b30-3f30-4718-99e3-7144caa7da9b" containerName="revoke-keys"
Oct 29 22:32:25.625432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.625373    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8f370020-4408-4381-9edd-9ecfd8228ad8" containerName="revoke-keys"
Oct 29 22:32:25.625432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.625378    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f370020-4408-4381-9edd-9ecfd8228ad8" containerName="revoke-keys"
Oct 29 22:32:25.625542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.625491    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe55b30-3f30-4718-99e3-7144caa7da9b" containerName="revoke-keys"
Oct 29 22:32:25.625542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.625501    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f370020-4408-4381-9edd-9ecfd8228ad8" containerName="revoke-keys"
Oct 29 22:32:25.625542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.625513    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a71204-5e6e-4ee4-aae2-6daeabe95b7c" containerName="revoke-keys"
Oct 29 22:32:25.630698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.630679    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:25.647077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.647056    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"]
Oct 29 22:32:25.653407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.653383    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:32:25.653519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.653413    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:32:25.661290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.661267    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:32:25.661290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.661286    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:32:25.726201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.726170    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04c8a962-8899s\" (UID: \"fb3272a7-2bc8-4740-8de4-f34616108e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:25.726373 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.726296    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04c8a962-8899s\" (UID: \"fb3272a7-2bc8-4740-8de4-f34616108e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:25.726373 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.726357    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhh4q\" (UniqueName: \"kubernetes.io/projected/fb3272a7-2bc8-4740-8de4-f34616108e39-kube-api-access-mhh4q\") pod \"maas-api-revoke-sub-04c8a962-8899s\" (UID: \"fb3272a7-2bc8-4740-8de4-f34616108e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:25.827876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.827840    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhh4q\" (UniqueName: \"kubernetes.io/projected/fb3272a7-2bc8-4740-8de4-f34616108e39-kube-api-access-mhh4q\") pod \"maas-api-revoke-sub-04c8a962-8899s\" (UID: \"fb3272a7-2bc8-4740-8de4-f34616108e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:25.828054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.828038    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04c8a962-8899s\" (UID: \"fb3272a7-2bc8-4740-8de4-f34616108e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:25.828119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.828107    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04c8a962-8899s\" (UID: \"fb3272a7-2bc8-4740-8de4-f34616108e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:25.828732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.828710    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04c8a962-8899s\" (UID: \"fb3272a7-2bc8-4740-8de4-f34616108e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:25.830610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.830588    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04c8a962-8899s\" (UID: \"fb3272a7-2bc8-4740-8de4-f34616108e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:25.851519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.851498    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhh4q\" (UniqueName: \"kubernetes.io/projected/fb3272a7-2bc8-4740-8de4-f34616108e39-kube-api-access-mhh4q\") pod \"maas-api-revoke-sub-04c8a962-8899s\" (UID: \"fb3272a7-2bc8-4740-8de4-f34616108e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:25.946803 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:25.946785    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:26.251644 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:26.251614    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3272a7_2bc8_4740_8de4_f34616108e39.slice/crio-094feff58baefd4d074294d02ebf153b1725c443402f9bbdc9fc12c664886d92 WatchSource:0}: Error finding container 094feff58baefd4d074294d02ebf153b1725c443402f9bbdc9fc12c664886d92: Status 404 returned error can't find the container with id 094feff58baefd4d074294d02ebf153b1725c443402f9bbdc9fc12c664886d92
Oct 29 22:32:26.256258 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:26.256237    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"]
Oct 29 22:32:26.619028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:26.618993    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s" event={"ID":"fb3272a7-2bc8-4740-8de4-f34616108e39","Type":"ContainerStarted","Data":"de5084434985ce3fdcb924d549ffb3c6e7c787035045a92e356a9120c2aade58"}
Oct 29 22:32:26.619178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:26.619048    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s" event={"ID":"fb3272a7-2bc8-4740-8de4-f34616108e39","Type":"ContainerStarted","Data":"094feff58baefd4d074294d02ebf153b1725c443402f9bbdc9fc12c664886d92"}
Oct 29 22:32:26.661727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:26.661673    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s" podStartSLOduration=1.661661276 podStartE2EDuration="1.661661276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:26.659432403 +0000 UTC m=+1088.684208546" watchObservedRunningTime="2026-10-29 22:32:26.661661276 +0000 UTC m=+1088.686437421"
Oct 29 22:32:27.026673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.026637    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:27.299276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.299191    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"]
Oct 29 22:32:27.304639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.304617    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:27.322361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.322319    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"]
Oct 29 22:32:27.450888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.450857    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25de7e1-x69zq\" (UID: \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:27.451052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.451013    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zvd\" (UniqueName: \"kubernetes.io/projected/e6f28939-d1c6-48db-b86b-01bea1e8a15b-kube-api-access-d9zvd\") pod \"maas-api-revoke-sub-f25de7e1-x69zq\" (UID: \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:27.451162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.451149    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25de7e1-x69zq\" (UID: \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:27.552673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.552597    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25de7e1-x69zq\" (UID: \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:27.552673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.552652    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25de7e1-x69zq\" (UID: \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:27.552840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.552789    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zvd\" (UniqueName: \"kubernetes.io/projected/e6f28939-d1c6-48db-b86b-01bea1e8a15b-kube-api-access-d9zvd\") pod \"maas-api-revoke-sub-f25de7e1-x69zq\" (UID: \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:27.553278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.553257    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25de7e1-x69zq\" (UID: \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:27.555000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.554981    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25de7e1-x69zq\" (UID: \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:27.573335 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.573296    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zvd\" (UniqueName: \"kubernetes.io/projected/e6f28939-d1c6-48db-b86b-01bea1e8a15b-kube-api-access-d9zvd\") pod \"maas-api-revoke-sub-f25de7e1-x69zq\" (UID: \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:27.629454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.629420    4860 generic.go:356] "Generic (PLEG): container finished" podID="fb3272a7-2bc8-4740-8de4-f34616108e39" containerID="de5084434985ce3fdcb924d549ffb3c6e7c787035045a92e356a9120c2aade58" exitCode=0
Oct 29 22:32:27.629582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.629454    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s" event={"ID":"fb3272a7-2bc8-4740-8de4-f34616108e39","Type":"ContainerDied","Data":"de5084434985ce3fdcb924d549ffb3c6e7c787035045a92e356a9120c2aade58"}
Oct 29 22:32:27.629639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.629612    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:27.933677 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:27.933648    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f28939_d1c6_48db_b86b_01bea1e8a15b.slice/crio-1d7459b35de76c1e0dc5de52bf24b380138e25903baa3310fd4188bd3d0d4895 WatchSource:0}: Error finding container 1d7459b35de76c1e0dc5de52bf24b380138e25903baa3310fd4188bd3d0d4895: Status 404 returned error can't find the container with id 1d7459b35de76c1e0dc5de52bf24b380138e25903baa3310fd4188bd3d0d4895
Oct 29 22:32:27.939402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:27.939378    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"]
Oct 29 22:32:28.643210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:28.643131    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq" event={"ID":"e6f28939-d1c6-48db-b86b-01bea1e8a15b","Type":"ContainerStarted","Data":"8f6151214a5bc27d8cd48d00bc6551293620e3e66c87f963f7a9a98eea0e3b4a"}
Oct 29 22:32:28.643629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:28.643221    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq" event={"ID":"e6f28939-d1c6-48db-b86b-01bea1e8a15b","Type":"ContainerStarted","Data":"1d7459b35de76c1e0dc5de52bf24b380138e25903baa3310fd4188bd3d0d4895"}
Oct 29 22:32:28.668989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:28.668957    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:28.692656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:28.692604    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq" podStartSLOduration=1.692585018 podStartE2EDuration="1.692585018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:28.686005124 +0000 UTC m=+1090.710781255" watchObservedRunningTime="2026-10-29 22:32:28.692585018 +0000 UTC m=+1090.717361149"
Oct 29 22:32:28.872461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:28.872431    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:28.978491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:28.978440    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-openshift-ca\") pod \"fb3272a7-2bc8-4740-8de4-f34616108e39\" (UID: \"fb3272a7-2bc8-4740-8de4-f34616108e39\") "
Oct 29 22:32:28.978649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:28.978633    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-xks-ca\") pod \"fb3272a7-2bc8-4740-8de4-f34616108e39\" (UID: \"fb3272a7-2bc8-4740-8de4-f34616108e39\") "
Oct 29 22:32:28.978729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:28.978709    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb3272a7-2bc8-4740-8de4-f34616108e39-kube-api-access-mhh4q\" (UniqueName: \"kubernetes.io/projected/fb3272a7-2bc8-4740-8de4-f34616108e39-kube-api-access-mhh4q\") pod \"fb3272a7-2bc8-4740-8de4-f34616108e39\" (UID: \"fb3272a7-2bc8-4740-8de4-f34616108e39\") "
Oct 29 22:32:28.978822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:28.978791    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-openshift-ca" pod "fb3272a7-2bc8-4740-8de4-f34616108e39" (UID: "fb3272a7-2bc8-4740-8de4-f34616108e39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:28.979248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:28.979226    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:28.981474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:28.981447    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3272a7-2bc8-4740-8de4-f34616108e39-kube-api-access-mhh4q" pod "fb3272a7-2bc8-4740-8de4-f34616108e39" (UID: "fb3272a7-2bc8-4740-8de4-f34616108e39"). InnerVolumeSpecName "kube-api-access-mhh4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:28.981650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:28.981635    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-xks-ca" pod "fb3272a7-2bc8-4740-8de4-f34616108e39" (UID: "fb3272a7-2bc8-4740-8de4-f34616108e39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:29.080451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.080428    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3272a7-2bc8-4740-8de4-f34616108e39-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:29.080451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.080452    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhh4q\" (UniqueName: \"kubernetes.io/projected/fb3272a7-2bc8-4740-8de4-f34616108e39-kube-api-access-mhh4q\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:29.203546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.203518    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"]
Oct 29 22:32:29.204151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.204137    4860 cpu_manager.go:414] "containerMap: removing container" podUID="fb3272a7-2bc8-4740-8de4-f34616108e39" containerName="revoke-keys"
Oct 29 22:32:29.204204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.204153    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb3272a7-2bc8-4740-8de4-f34616108e39" containerName="revoke-keys"
Oct 29 22:32:29.204276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.204267    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb3272a7-2bc8-4740-8de4-f34616108e39" containerName="revoke-keys"
Oct 29 22:32:29.208168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.208154    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:29.226219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.226198    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"]
Oct 29 22:32:29.282659 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.282630    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-562xp\" (UniqueName: \"kubernetes.io/projected/9ccb50ac-9a19-4084-a015-eafd58a59202-kube-api-access-562xp\") pod \"maas-api-revoke-sub-47a7c2ff-mfztr\" (UID: \"9ccb50ac-9a19-4084-a015-eafd58a59202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:29.282797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.282773    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a7c2ff-mfztr\" (UID: \"9ccb50ac-9a19-4084-a015-eafd58a59202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:29.282849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.282836    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a7c2ff-mfztr\" (UID: \"9ccb50ac-9a19-4084-a015-eafd58a59202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:29.384275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.384248    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-562xp\" (UniqueName: \"kubernetes.io/projected/9ccb50ac-9a19-4084-a015-eafd58a59202-kube-api-access-562xp\") pod \"maas-api-revoke-sub-47a7c2ff-mfztr\" (UID: \"9ccb50ac-9a19-4084-a015-eafd58a59202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:29.384418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.384384    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a7c2ff-mfztr\" (UID: \"9ccb50ac-9a19-4084-a015-eafd58a59202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:29.384457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.384428    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a7c2ff-mfztr\" (UID: \"9ccb50ac-9a19-4084-a015-eafd58a59202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:29.385145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.385121    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a7c2ff-mfztr\" (UID: \"9ccb50ac-9a19-4084-a015-eafd58a59202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:29.386934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.386910    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a7c2ff-mfztr\" (UID: \"9ccb50ac-9a19-4084-a015-eafd58a59202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:29.405661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.405638    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-562xp\" (UniqueName: \"kubernetes.io/projected/9ccb50ac-9a19-4084-a015-eafd58a59202-kube-api-access-562xp\") pod \"maas-api-revoke-sub-47a7c2ff-mfztr\" (UID: \"9ccb50ac-9a19-4084-a015-eafd58a59202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:29.522863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.522793    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:29.654027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.654001    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"
Oct 29 22:32:29.654527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.654027    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s" event={"ID":"fb3272a7-2bc8-4740-8de4-f34616108e39","Type":"ContainerDied","Data":"094feff58baefd4d074294d02ebf153b1725c443402f9bbdc9fc12c664886d92"}
Oct 29 22:32:29.654527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.654073    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094feff58baefd4d074294d02ebf153b1725c443402f9bbdc9fc12c664886d92"
Oct 29 22:32:29.658071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.658044    4860 generic.go:356] "Generic (PLEG): container finished" podID="e6f28939-d1c6-48db-b86b-01bea1e8a15b" containerID="8f6151214a5bc27d8cd48d00bc6551293620e3e66c87f963f7a9a98eea0e3b4a" exitCode=0
Oct 29 22:32:29.658276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.658257    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq" event={"ID":"e6f28939-d1c6-48db-b86b-01bea1e8a15b","Type":"ContainerDied","Data":"8f6151214a5bc27d8cd48d00bc6551293620e3e66c87f963f7a9a98eea0e3b4a"}
Oct 29 22:32:29.832985 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:29.832238    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccb50ac_9a19_4084_a015_eafd58a59202.slice/crio-8869e4af3c0349591fb0417e2d298b5915a94ee25a13bc717a1d5fa9ece41df7 WatchSource:0}: Error finding container 8869e4af3c0349591fb0417e2d298b5915a94ee25a13bc717a1d5fa9ece41df7: Status 404 returned error can't find the container with id 8869e4af3c0349591fb0417e2d298b5915a94ee25a13bc717a1d5fa9ece41df7
Oct 29 22:32:29.835988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:29.835963    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"]
Oct 29 22:32:30.332996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:30.332910    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:30.668138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:30.668053    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr" event={"ID":"9ccb50ac-9a19-4084-a015-eafd58a59202","Type":"ContainerStarted","Data":"7c9c6c0976e363cb89a2b918b75b6fe729c10cd824955bfe401d221c037061bc"}
Oct 29 22:32:30.668138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:30.668110    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr" event={"ID":"9ccb50ac-9a19-4084-a015-eafd58a59202","Type":"ContainerStarted","Data":"8869e4af3c0349591fb0417e2d298b5915a94ee25a13bc717a1d5fa9ece41df7"}
Oct 29 22:32:30.710597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:30.710536    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr" podStartSLOduration=1.7105218899999999 podStartE2EDuration="1.71052189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:30.70947853 +0000 UTC m=+1092.734254662" watchObservedRunningTime="2026-10-29 22:32:30.71052189 +0000 UTC m=+1092.735298021"
Oct 29 22:32:30.909508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:30.909483    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:31.006808 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.006776    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-openshift-ca\") pod \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\" (UID: \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\") "
Oct 29 22:32:31.006956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.006824    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f28939-d1c6-48db-b86b-01bea1e8a15b-kube-api-access-d9zvd\" (UniqueName: \"kubernetes.io/projected/e6f28939-d1c6-48db-b86b-01bea1e8a15b-kube-api-access-d9zvd\") pod \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\" (UID: \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\") "
Oct 29 22:32:31.006956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.006929    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-xks-ca\") pod \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\" (UID: \"e6f28939-d1c6-48db-b86b-01bea1e8a15b\") "
Oct 29 22:32:31.007207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.007184    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-openshift-ca" pod "e6f28939-d1c6-48db-b86b-01bea1e8a15b" (UID: "e6f28939-d1c6-48db-b86b-01bea1e8a15b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:31.007553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.007537    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:31.009000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.008978    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-xks-ca" pod "e6f28939-d1c6-48db-b86b-01bea1e8a15b" (UID: "e6f28939-d1c6-48db-b86b-01bea1e8a15b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:31.009053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.009032    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f28939-d1c6-48db-b86b-01bea1e8a15b-kube-api-access-d9zvd" pod "e6f28939-d1c6-48db-b86b-01bea1e8a15b" (UID: "e6f28939-d1c6-48db-b86b-01bea1e8a15b"). InnerVolumeSpecName "kube-api-access-d9zvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:31.109200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.109176    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f28939-d1c6-48db-b86b-01bea1e8a15b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:31.109200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.109201    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9zvd\" (UniqueName: \"kubernetes.io/projected/e6f28939-d1c6-48db-b86b-01bea1e8a15b-kube-api-access-d9zvd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:31.670161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.670120    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:31.678114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.678089    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"
Oct 29 22:32:31.678248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.678106    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq" event={"ID":"e6f28939-d1c6-48db-b86b-01bea1e8a15b","Type":"ContainerDied","Data":"1d7459b35de76c1e0dc5de52bf24b380138e25903baa3310fd4188bd3d0d4895"}
Oct 29 22:32:31.678248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.678178    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d7459b35de76c1e0dc5de52bf24b380138e25903baa3310fd4188bd3d0d4895"
Oct 29 22:32:31.680084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.680058    4860 generic.go:356] "Generic (PLEG): container finished" podID="9ccb50ac-9a19-4084-a015-eafd58a59202" containerID="7c9c6c0976e363cb89a2b918b75b6fe729c10cd824955bfe401d221c037061bc" exitCode=0
Oct 29 22:32:31.680175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.680106    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr" event={"ID":"9ccb50ac-9a19-4084-a015-eafd58a59202","Type":"ContainerDied","Data":"7c9c6c0976e363cb89a2b918b75b6fe729c10cd824955bfe401d221c037061bc"}
Oct 29 22:32:31.788861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.788830    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:31.914921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:31.914893    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:32.845483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:32.845447    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:32.848910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:32.848886    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:32.933465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:32.933437    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-xks-ca\") pod \"9ccb50ac-9a19-4084-a015-eafd58a59202\" (UID: \"9ccb50ac-9a19-4084-a015-eafd58a59202\") "
Oct 29 22:32:32.933683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:32.933666    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-openshift-ca\") pod \"9ccb50ac-9a19-4084-a015-eafd58a59202\" (UID: \"9ccb50ac-9a19-4084-a015-eafd58a59202\") "
Oct 29 22:32:32.933773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:32.933738    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccb50ac-9a19-4084-a015-eafd58a59202-kube-api-access-562xp\" (UniqueName: \"kubernetes.io/projected/9ccb50ac-9a19-4084-a015-eafd58a59202-kube-api-access-562xp\") pod \"9ccb50ac-9a19-4084-a015-eafd58a59202\" (UID: \"9ccb50ac-9a19-4084-a015-eafd58a59202\") "
Oct 29 22:32:32.934014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:32.933991    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-openshift-ca" pod "9ccb50ac-9a19-4084-a015-eafd58a59202" (UID: "9ccb50ac-9a19-4084-a015-eafd58a59202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:32.934313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:32.934296    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:32.935748 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:32.935724    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-xks-ca" pod "9ccb50ac-9a19-4084-a015-eafd58a59202" (UID: "9ccb50ac-9a19-4084-a015-eafd58a59202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:32.935843 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:32.935793    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccb50ac-9a19-4084-a015-eafd58a59202-kube-api-access-562xp" pod "9ccb50ac-9a19-4084-a015-eafd58a59202" (UID: "9ccb50ac-9a19-4084-a015-eafd58a59202"). InnerVolumeSpecName "kube-api-access-562xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:33.036007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:33.035963    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-562xp\" (UniqueName: \"kubernetes.io/projected/9ccb50ac-9a19-4084-a015-eafd58a59202-kube-api-access-562xp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:33.036007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:33.036004    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ccb50ac-9a19-4084-a015-eafd58a59202-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:33.697707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:33.697677    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr" event={"ID":"9ccb50ac-9a19-4084-a015-eafd58a59202","Type":"ContainerDied","Data":"8869e4af3c0349591fb0417e2d298b5915a94ee25a13bc717a1d5fa9ece41df7"}
Oct 29 22:32:33.697707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:33.697693    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"
Oct 29 22:32:33.697915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:33.697731    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8869e4af3c0349591fb0417e2d298b5915a94ee25a13bc717a1d5fa9ece41df7"
Oct 29 22:32:33.827018 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:33.826983    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:35.779351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:35.775392    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:35.815350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:35.804395    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:35.927454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:35.927410    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.308466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.308419    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.309043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.309009    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:32:36.309462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.309442    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:32:36.329913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.329882    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Oct 29 22:32:36.330418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.330394    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded"
Oct 29 22:32:36.345419 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.345388    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:32:36.473500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.473453    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.529588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.529488    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:32:36.567568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.567456    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.596154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.596118    4860 patch_prober.go:29] interesting pod/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:32:36.596442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.596409    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.628351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.625413    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.779514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.779457    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.780879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.780855    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:32:36.813348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.809542    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:36.813348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.809952    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:32:36.966431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.966087    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:32:36.967883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:36.967093    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:32:37.309687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.309570    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:37.320952 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.320902    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:32:37.354351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.324424    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:32:37.354351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.324491    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:32:37.354351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.326541    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:37.354351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.347839    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:32:37.354351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.350422    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:37.354351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.350734    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Oct 29 22:32:37.354351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.351047    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:32:37.354351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.351102    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:32:37.484463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.479427    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:32:37.484463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.479783    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:32:37.571865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.538830    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:32:37.571865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.543441    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:32:37.644931 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.644899    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:32:37.671382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.646961    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:32:37.828401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.803482    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:32:37.932606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.932564    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:37.974515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:37.974473    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.346643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:38.346594    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:32:38.348056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:38.348015    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:32:38.351856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:38.351528    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.379348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:38.353492    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.379348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:38.353862    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:32:38.420466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:38.420410    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Oct 29 22:32:38.488416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:38.488373    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.559561 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:38.559516    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:38.658400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:38.657964    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.036348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.011178    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.039368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.039313    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.055013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.039829    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.246894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.246841    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:39.393649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.393555    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.406663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.406628    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.410188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.410161    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.431338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.414456    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.431843 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.414478    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Oct 29 22:32:39.521281 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:32:39.516928    4860 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.414s"
Oct 29 22:32:39.521839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.521818    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:32:39.522057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.522042    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:32:39.522216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.522202    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:32:39.578717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.578656    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"]
Oct 29 22:32:39.581568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.580207    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e6f28939-d1c6-48db-b86b-01bea1e8a15b" containerName="revoke-keys"
Oct 29 22:32:39.581790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.581771    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f28939-d1c6-48db-b86b-01bea1e8a15b" containerName="revoke-keys"
Oct 29 22:32:39.581938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.581926    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9ccb50ac-9a19-4084-a015-eafd58a59202" containerName="revoke-keys"
Oct 29 22:32:39.582026 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.582014    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccb50ac-9a19-4084-a015-eafd58a59202" containerName="revoke-keys"
Oct 29 22:32:39.582355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.582340    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ccb50ac-9a19-4084-a015-eafd58a59202" containerName="revoke-keys"
Oct 29 22:32:39.582484 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.582474    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f28939-d1c6-48db-b86b-01bea1e8a15b" containerName="revoke-keys"
Oct 29 22:32:39.587470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.587244    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.597611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.597591    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:39.621587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.620749    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:32:39.621587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.620916    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:32:39.621587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.620998    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:32:39.621587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.621083    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:32:39.628387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.628365    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"]
Oct 29 22:32:39.647905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.647836    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:32:39.667575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.667524    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.702226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.699758    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:32:39.730346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.725217    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htd46\" (UniqueName: \"kubernetes.io/projected/6095b303-414b-4f09-94fe-99216332dff3-kube-api-access-htd46\") pod \"maas-api-revoke-sub-37b369ae-zs2kc\" (UID: \"6095b303-414b-4f09-94fe-99216332dff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:39.730346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.725270    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6095b303-414b-4f09-94fe-99216332dff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b369ae-zs2kc\" (UID: \"6095b303-414b-4f09-94fe-99216332dff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:39.730346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.725485    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6095b303-414b-4f09-94fe-99216332dff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b369ae-zs2kc\" (UID: \"6095b303-414b-4f09-94fe-99216332dff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:39.730346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.725632    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"]
Oct 29 22:32:39.751611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.751589    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:39.851910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.851859    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htd46\" (UniqueName: \"kubernetes.io/projected/6095b303-414b-4f09-94fe-99216332dff3-kube-api-access-htd46\") pod \"maas-api-revoke-sub-37b369ae-zs2kc\" (UID: \"6095b303-414b-4f09-94fe-99216332dff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:39.852159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.852139    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6095b303-414b-4f09-94fe-99216332dff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b369ae-zs2kc\" (UID: \"6095b303-414b-4f09-94fe-99216332dff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:39.852523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.852496    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6095b303-414b-4f09-94fe-99216332dff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b369ae-zs2kc\" (UID: \"6095b303-414b-4f09-94fe-99216332dff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:39.853644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.853619    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6095b303-414b-4f09-94fe-99216332dff3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b369ae-zs2kc\" (UID: \"6095b303-414b-4f09-94fe-99216332dff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:39.855041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.855021    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:32:39.864362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.862867    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:32:39.869246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.869220    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6095b303-414b-4f09-94fe-99216332dff3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b369ae-zs2kc\" (UID: \"6095b303-414b-4f09-94fe-99216332dff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:39.902141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.883931    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"]
Oct 29 22:32:39.953602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.951947    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htd46\" (UniqueName: \"kubernetes.io/projected/6095b303-414b-4f09-94fe-99216332dff3-kube-api-access-htd46\") pod \"maas-api-revoke-sub-37b369ae-zs2kc\" (UID: \"6095b303-414b-4f09-94fe-99216332dff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:39.953602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.952561    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:39.953894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.953865    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlnnv\" (UniqueName: \"kubernetes.io/projected/7def8e50-a52d-4771-9d3a-3cef50e4e5af-kube-api-access-qlnnv\") pod \"maas-api-revoke-sub-8578faa4-kxmjg\" (UID: \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:39.954524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.954485    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8578faa4-kxmjg\" (UID: \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:39.954692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.954673    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8578faa4-kxmjg\" (UID: \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:39.957833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.957816    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:32:39.991401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:39.991376    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:32:40.015489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.014261    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:32:40.050405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.048705    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:32:40.052925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.052905    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:32:40.055664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.055642    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlnnv\" (UniqueName: \"kubernetes.io/projected/7def8e50-a52d-4771-9d3a-3cef50e4e5af-kube-api-access-qlnnv\") pod \"maas-api-revoke-sub-8578faa4-kxmjg\" (UID: \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:40.055768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.055725    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8578faa4-kxmjg\" (UID: \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:40.055768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.055763    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8578faa4-kxmjg\" (UID: \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:40.058641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.058615    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8578faa4-kxmjg\" (UID: \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:40.059196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.059177    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8578faa4-kxmjg\" (UID: \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:40.076827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.076656    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlnnv\" (UniqueName: \"kubernetes.io/projected/7def8e50-a52d-4771-9d3a-3cef50e4e5af-kube-api-access-qlnnv\") pod \"maas-api-revoke-sub-8578faa4-kxmjg\" (UID: \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:40.224486 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.224454    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:40.341227 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:40.341167    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6095b303_414b_4f09_94fe_99216332dff3.slice/crio-3a2ecc65d1c440541aae944b0dd7a05c976cdc35d847ac8a81187e6ca0b61d13 WatchSource:0}: Error finding container 3a2ecc65d1c440541aae944b0dd7a05c976cdc35d847ac8a81187e6ca0b61d13: Status 404 returned error can't find the container with id 3a2ecc65d1c440541aae944b0dd7a05c976cdc35d847ac8a81187e6ca0b61d13
Oct 29 22:32:40.346797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.346772    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"]
Oct 29 22:32:40.383810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.380756    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:40.536163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:40.536132    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"]
Oct 29 22:32:40.571497 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:40.571466    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7def8e50_a52d_4771_9d3a_3cef50e4e5af.slice/crio-be17d97e8145517cfbcb6270093424e7971334ba01cbe9d40385751207186848 WatchSource:0}: Error finding container be17d97e8145517cfbcb6270093424e7971334ba01cbe9d40385751207186848: Status 404 returned error can't find the container with id be17d97e8145517cfbcb6270093424e7971334ba01cbe9d40385751207186848
Oct 29 22:32:41.266889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:41.266858    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg" event={"ID":"7def8e50-a52d-4771-9d3a-3cef50e4e5af","Type":"ContainerStarted","Data":"165f414f65fb13c90d6353ac338c84ebe4de3709ba8d9a66a425787fd4516d41"}
Oct 29 22:32:41.267387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:41.266907    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg" event={"ID":"7def8e50-a52d-4771-9d3a-3cef50e4e5af","Type":"ContainerStarted","Data":"be17d97e8145517cfbcb6270093424e7971334ba01cbe9d40385751207186848"}
Oct 29 22:32:41.268651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:41.268619    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc" event={"ID":"6095b303-414b-4f09-94fe-99216332dff3","Type":"ContainerStarted","Data":"fd149bfcdc3d624f9335a375e233919b96d7a65b924156d2320a7914be8db78b"}
Oct 29 22:32:41.268750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:41.268700    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc" event={"ID":"6095b303-414b-4f09-94fe-99216332dff3","Type":"ContainerStarted","Data":"3a2ecc65d1c440541aae944b0dd7a05c976cdc35d847ac8a81187e6ca0b61d13"}
Oct 29 22:32:41.307472 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:41.307429    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg" podStartSLOduration=2.307417604 podStartE2EDuration="2.307417604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:41.306464739 +0000 UTC m=+1103.331240867" watchObservedRunningTime="2026-10-29 22:32:41.307417604 +0000 UTC m=+1103.332193735"
Oct 29 22:32:41.348053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:41.348005    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc" podStartSLOduration=3.347990281 podStartE2EDuration="3.347990281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:41.345773418 +0000 UTC m=+1103.370549548" watchObservedRunningTime="2026-10-29 22:32:41.347990281 +0000 UTC m=+1103.372766416"
Oct 29 22:32:42.053633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:42.053599    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:42.280430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:42.280389    4860 generic.go:356] "Generic (PLEG): container finished" podID="7def8e50-a52d-4771-9d3a-3cef50e4e5af" containerID="165f414f65fb13c90d6353ac338c84ebe4de3709ba8d9a66a425787fd4516d41" exitCode=0
Oct 29 22:32:42.280918 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:42.280485    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg" event={"ID":"7def8e50-a52d-4771-9d3a-3cef50e4e5af","Type":"ContainerDied","Data":"165f414f65fb13c90d6353ac338c84ebe4de3709ba8d9a66a425787fd4516d41"}
Oct 29 22:32:42.282588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:42.282566    4860 generic.go:356] "Generic (PLEG): container finished" podID="6095b303-414b-4f09-94fe-99216332dff3" containerID="fd149bfcdc3d624f9335a375e233919b96d7a65b924156d2320a7914be8db78b" exitCode=0
Oct 29 22:32:42.282738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:42.282608    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc" event={"ID":"6095b303-414b-4f09-94fe-99216332dff3","Type":"ContainerDied","Data":"fd149bfcdc3d624f9335a375e233919b96d7a65b924156d2320a7914be8db78b"}
Oct 29 22:32:43.100219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.100186    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"]
Oct 29 22:32:43.100438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.100412    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" podUID="4761326c-38e8-466e-8946-3c75605d0ad7" containerName="istio-proxy" containerID="cri-o://956f3c1df683e571a0088062426798ed37c6fe83c31684139b60188912b5cfbb" gracePeriod=30
Oct 29 22:32:43.496497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.496469    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:43.518447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.518424    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-openshift-ca\") pod \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\" (UID: \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\") "
Oct 29 22:32:43.518582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.518549    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-xks-ca\") pod \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\" (UID: \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\") "
Oct 29 22:32:43.518710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.518696    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7def8e50-a52d-4771-9d3a-3cef50e4e5af-kube-api-access-qlnnv\" (UniqueName: \"kubernetes.io/projected/7def8e50-a52d-4771-9d3a-3cef50e4e5af-kube-api-access-qlnnv\") pod \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\" (UID: \"7def8e50-a52d-4771-9d3a-3cef50e4e5af\") "
Oct 29 22:32:43.518772 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.518728    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-openshift-ca" pod "7def8e50-a52d-4771-9d3a-3cef50e4e5af" (UID: "7def8e50-a52d-4771-9d3a-3cef50e4e5af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:43.519231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.519212    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:43.520566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.520545    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-xks-ca" pod "7def8e50-a52d-4771-9d3a-3cef50e4e5af" (UID: "7def8e50-a52d-4771-9d3a-3cef50e4e5af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:43.520797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.520772    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7def8e50-a52d-4771-9d3a-3cef50e4e5af-kube-api-access-qlnnv" pod "7def8e50-a52d-4771-9d3a-3cef50e4e5af" (UID: "7def8e50-a52d-4771-9d3a-3cef50e4e5af"). InnerVolumeSpecName "kube-api-access-qlnnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:43.542072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.542045    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:43.547723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.547704    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:43.621838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.621754    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6095b303-414b-4f09-94fe-99216332dff3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6095b303-414b-4f09-94fe-99216332dff3-maas-api-openshift-ca\") pod \"6095b303-414b-4f09-94fe-99216332dff3\" (UID: \"6095b303-414b-4f09-94fe-99216332dff3\") "
Oct 29 22:32:43.622014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.621985    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6095b303-414b-4f09-94fe-99216332dff3-kube-api-access-htd46\" (UniqueName: \"kubernetes.io/projected/6095b303-414b-4f09-94fe-99216332dff3-kube-api-access-htd46\") pod \"6095b303-414b-4f09-94fe-99216332dff3\" (UID: \"6095b303-414b-4f09-94fe-99216332dff3\") "
Oct 29 22:32:43.622111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.622092    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6095b303-414b-4f09-94fe-99216332dff3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6095b303-414b-4f09-94fe-99216332dff3-maas-api-xks-ca\") pod \"6095b303-414b-4f09-94fe-99216332dff3\" (UID: \"6095b303-414b-4f09-94fe-99216332dff3\") "
Oct 29 22:32:43.622180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.622159    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6095b303-414b-4f09-94fe-99216332dff3-maas-api-openshift-ca" pod "6095b303-414b-4f09-94fe-99216332dff3" (UID: "6095b303-414b-4f09-94fe-99216332dff3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:43.622759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.622736    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6095b303-414b-4f09-94fe-99216332dff3-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:43.622759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.622760    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7def8e50-a52d-4771-9d3a-3cef50e4e5af-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:43.622977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.622769    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlnnv\" (UniqueName: \"kubernetes.io/projected/7def8e50-a52d-4771-9d3a-3cef50e4e5af-kube-api-access-qlnnv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:43.625259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.625227    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6095b303-414b-4f09-94fe-99216332dff3-kube-api-access-htd46" pod "6095b303-414b-4f09-94fe-99216332dff3" (UID: "6095b303-414b-4f09-94fe-99216332dff3"). InnerVolumeSpecName "kube-api-access-htd46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:43.626172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.626139    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6095b303-414b-4f09-94fe-99216332dff3-maas-api-xks-ca" pod "6095b303-414b-4f09-94fe-99216332dff3" (UID: "6095b303-414b-4f09-94fe-99216332dff3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:43.634294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.634263    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"]
Oct 29 22:32:43.635138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.635098    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6095b303-414b-4f09-94fe-99216332dff3" containerName="revoke-keys"
Oct 29 22:32:43.635138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.635119    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6095b303-414b-4f09-94fe-99216332dff3" containerName="revoke-keys"
Oct 29 22:32:43.635317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.635205    4860 cpu_manager.go:414] "containerMap: removing container" podUID="7def8e50-a52d-4771-9d3a-3cef50e4e5af" containerName="revoke-keys"
Oct 29 22:32:43.635317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.635215    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7def8e50-a52d-4771-9d3a-3cef50e4e5af" containerName="revoke-keys"
Oct 29 22:32:43.635499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.635483    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7def8e50-a52d-4771-9d3a-3cef50e4e5af" containerName="revoke-keys"
Oct 29 22:32:43.635562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.635509    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6095b303-414b-4f09-94fe-99216332dff3" containerName="revoke-keys"
Oct 29 22:32:43.639913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.639893    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:43.650920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.650662    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"]
Oct 29 22:32:43.723496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.723450    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47b2j\" (UniqueName: \"kubernetes.io/projected/6266e819-5a4a-4dd2-a11c-b008142baf37-kube-api-access-47b2j\") pod \"maas-api-revoke-sub-2d81d613-2sh9x\" (UID: \"6266e819-5a4a-4dd2-a11c-b008142baf37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:43.723496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.723500    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d81d613-2sh9x\" (UID: \"6266e819-5a4a-4dd2-a11c-b008142baf37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:43.723737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.723654    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d81d613-2sh9x\" (UID: \"6266e819-5a4a-4dd2-a11c-b008142baf37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:43.723947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.723928    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htd46\" (UniqueName: \"kubernetes.io/projected/6095b303-414b-4f09-94fe-99216332dff3-kube-api-access-htd46\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:43.724011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.723954    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6095b303-414b-4f09-94fe-99216332dff3-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:43.825651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.825625    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47b2j\" (UniqueName: \"kubernetes.io/projected/6266e819-5a4a-4dd2-a11c-b008142baf37-kube-api-access-47b2j\") pod \"maas-api-revoke-sub-2d81d613-2sh9x\" (UID: \"6266e819-5a4a-4dd2-a11c-b008142baf37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:43.825822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.825666    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d81d613-2sh9x\" (UID: \"6266e819-5a4a-4dd2-a11c-b008142baf37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:43.825822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.825774    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d81d613-2sh9x\" (UID: \"6266e819-5a4a-4dd2-a11c-b008142baf37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:43.826387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.826366    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d81d613-2sh9x\" (UID: \"6266e819-5a4a-4dd2-a11c-b008142baf37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:43.828123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.828104    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d81d613-2sh9x\" (UID: \"6266e819-5a4a-4dd2-a11c-b008142baf37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:43.846886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.846864    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47b2j\" (UniqueName: \"kubernetes.io/projected/6266e819-5a4a-4dd2-a11c-b008142baf37-kube-api-access-47b2j\") pod \"maas-api-revoke-sub-2d81d613-2sh9x\" (UID: \"6266e819-5a4a-4dd2-a11c-b008142baf37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:43.959277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:43.959254    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:44.261853 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:44.261826    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6266e819_5a4a_4dd2_a11c_b008142baf37.slice/crio-7abdf7a527e611b8808f8c2a5ecf52a377d9cab8b439721c3b1dcabe39fdef8f WatchSource:0}: Error finding container 7abdf7a527e611b8808f8c2a5ecf52a377d9cab8b439721c3b1dcabe39fdef8f: Status 404 returned error can't find the container with id 7abdf7a527e611b8808f8c2a5ecf52a377d9cab8b439721c3b1dcabe39fdef8f
Oct 29 22:32:44.266040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:44.266003    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"]
Oct 29 22:32:44.304850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:44.304824    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"
Oct 29 22:32:44.304963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:44.304850    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg" event={"ID":"7def8e50-a52d-4771-9d3a-3cef50e4e5af","Type":"ContainerDied","Data":"be17d97e8145517cfbcb6270093424e7971334ba01cbe9d40385751207186848"}
Oct 29 22:32:44.304963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:44.304895    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be17d97e8145517cfbcb6270093424e7971334ba01cbe9d40385751207186848"
Oct 29 22:32:44.306310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:44.306293    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"
Oct 29 22:32:44.306310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:44.306303    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc" event={"ID":"6095b303-414b-4f09-94fe-99216332dff3","Type":"ContainerDied","Data":"3a2ecc65d1c440541aae944b0dd7a05c976cdc35d847ac8a81187e6ca0b61d13"}
Oct 29 22:32:44.306483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:44.306357    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a2ecc65d1c440541aae944b0dd7a05c976cdc35d847ac8a81187e6ca0b61d13"
Oct 29 22:32:44.307779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:44.307757    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x" event={"ID":"6266e819-5a4a-4dd2-a11c-b008142baf37","Type":"ContainerStarted","Data":"7abdf7a527e611b8808f8c2a5ecf52a377d9cab8b439721c3b1dcabe39fdef8f"}
Oct 29 22:32:45.218760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:45.218735    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:45.321522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:45.321491    4860 generic.go:356] "Generic (PLEG): container finished" podID="6266e819-5a4a-4dd2-a11c-b008142baf37" containerID="608d511a213d0ee5a64e1b2dcdd53cc413d02f5118022d82cd9b90141e48b12a" exitCode=0
Oct 29 22:32:45.321697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:45.321572    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x" event={"ID":"6266e819-5a4a-4dd2-a11c-b008142baf37","Type":"ContainerDied","Data":"608d511a213d0ee5a64e1b2dcdd53cc413d02f5118022d82cd9b90141e48b12a"}
Oct 29 22:32:46.493877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:46.493852    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:46.559298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:46.559266    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-openshift-ca\") pod \"6266e819-5a4a-4dd2-a11c-b008142baf37\" (UID: \"6266e819-5a4a-4dd2-a11c-b008142baf37\") "
Oct 29 22:32:46.559461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:46.559362    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6266e819-5a4a-4dd2-a11c-b008142baf37-kube-api-access-47b2j\" (UniqueName: \"kubernetes.io/projected/6266e819-5a4a-4dd2-a11c-b008142baf37-kube-api-access-47b2j\") pod \"6266e819-5a4a-4dd2-a11c-b008142baf37\" (UID: \"6266e819-5a4a-4dd2-a11c-b008142baf37\") "
Oct 29 22:32:46.559543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:46.559525    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-xks-ca\") pod \"6266e819-5a4a-4dd2-a11c-b008142baf37\" (UID: \"6266e819-5a4a-4dd2-a11c-b008142baf37\") "
Oct 29 22:32:46.559747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:46.559719    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-openshift-ca" pod "6266e819-5a4a-4dd2-a11c-b008142baf37" (UID: "6266e819-5a4a-4dd2-a11c-b008142baf37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:46.560209 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:46.560186    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:46.562036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:46.562000    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6266e819-5a4a-4dd2-a11c-b008142baf37-kube-api-access-47b2j" pod "6266e819-5a4a-4dd2-a11c-b008142baf37" (UID: "6266e819-5a4a-4dd2-a11c-b008142baf37"). InnerVolumeSpecName "kube-api-access-47b2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:46.562156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:46.562058    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-xks-ca" pod "6266e819-5a4a-4dd2-a11c-b008142baf37" (UID: "6266e819-5a4a-4dd2-a11c-b008142baf37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:46.661222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:46.661169    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47b2j\" (UniqueName: \"kubernetes.io/projected/6266e819-5a4a-4dd2-a11c-b008142baf37-kube-api-access-47b2j\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:46.661222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:46.661193    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6266e819-5a4a-4dd2-a11c-b008142baf37-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:46.861570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:46.861537    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:47.348712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:47.348684    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"
Oct 29 22:32:47.348712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:47.348686    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x" event={"ID":"6266e819-5a4a-4dd2-a11c-b008142baf37","Type":"ContainerDied","Data":"7abdf7a527e611b8808f8c2a5ecf52a377d9cab8b439721c3b1dcabe39fdef8f"}
Oct 29 22:32:47.348962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:47.348791    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7abdf7a527e611b8808f8c2a5ecf52a377d9cab8b439721c3b1dcabe39fdef8f"
Oct 29 22:32:48.060021 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:32:48.059985    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:32:48.367926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.367898    4860 generic.go:356] "Generic (PLEG): container finished" podID="4761326c-38e8-466e-8946-3c75605d0ad7" containerID="956f3c1df683e571a0088062426798ed37c6fe83c31684139b60188912b5cfbb" exitCode=0
Oct 29 22:32:48.368061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.367987    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" event={"ID":"4761326c-38e8-466e-8946-3c75605d0ad7","Type":"ContainerDied","Data":"956f3c1df683e571a0088062426798ed37c6fe83c31684139b60188912b5cfbb"}
Oct 29 22:32:48.368061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.368040    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5" event={"ID":"4761326c-38e8-466e-8946-3c75605d0ad7","Type":"ContainerDied","Data":"fbe26575d84ace02e9c1c6950e15a9fabf0c803c8324b142d718f27f2ac752e5"}
Oct 29 22:32:48.368209 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.368063    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbe26575d84ace02e9c1c6950e15a9fabf0c803c8324b142d718f27f2ac752e5"
Oct 29 22:32:48.382653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.382631    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:32:48.393030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.393010    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:48.484599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.484558    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-istio-token\" (UniqueName: \"kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-istio-token\") pod \"4761326c-38e8-466e-8946-3c75605d0ad7\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") "
Oct 29 22:32:48.484764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.484619    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-credential-socket\") pod \"4761326c-38e8-466e-8946-3c75605d0ad7\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") "
Oct 29 22:32:48.484764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.484659    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4761326c-38e8-466e-8946-3c75605d0ad7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4761326c-38e8-466e-8946-3c75605d0ad7-istiod-ca-cert\") pod \"4761326c-38e8-466e-8946-3c75605d0ad7\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") "
Oct 29 22:32:48.484764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.484696    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-certs\") pod \"4761326c-38e8-466e-8946-3c75605d0ad7\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") "
Oct 29 22:32:48.484764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.484743    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-data\") pod \"4761326c-38e8-466e-8946-3c75605d0ad7\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") "
Oct 29 22:32:48.484912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.484810    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-socket\") pod \"4761326c-38e8-466e-8946-3c75605d0ad7\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") "
Oct 29 22:32:48.484979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.484962    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-envoy\") pod \"4761326c-38e8-466e-8946-3c75605d0ad7\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") "
Oct 29 22:32:48.485487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.485054    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4761326c-38e8-466e-8946-3c75605d0ad7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4761326c-38e8-466e-8946-3c75605d0ad7-istio-podinfo\") pod \"4761326c-38e8-466e-8946-3c75605d0ad7\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") "
Oct 29 22:32:48.485487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.485201    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-kube-api-access-mt2dn\" (UniqueName: \"kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-kube-api-access-mt2dn\") pod \"4761326c-38e8-466e-8946-3c75605d0ad7\" (UID: \"4761326c-38e8-466e-8946-3c75605d0ad7\") "
Oct 29 22:32:48.486182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.486055    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-credential-socket" pod "4761326c-38e8-466e-8946-3c75605d0ad7" (UID: "4761326c-38e8-466e-8946-3c75605d0ad7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:48.486182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.486109    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-socket" pod "4761326c-38e8-466e-8946-3c75605d0ad7" (UID: "4761326c-38e8-466e-8946-3c75605d0ad7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:48.486374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.486348    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-certs" pod "4761326c-38e8-466e-8946-3c75605d0ad7" (UID: "4761326c-38e8-466e-8946-3c75605d0ad7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:48.486583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.486555    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-data" pod "4761326c-38e8-466e-8946-3c75605d0ad7" (UID: "4761326c-38e8-466e-8946-3c75605d0ad7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:48.486958 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.486927    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4761326c-38e8-466e-8946-3c75605d0ad7-istiod-ca-cert" pod "4761326c-38e8-466e-8946-3c75605d0ad7" (UID: "4761326c-38e8-466e-8946-3c75605d0ad7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:48.487211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.487191    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:48.487370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.487217    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4761326c-38e8-466e-8946-3c75605d0ad7-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:48.487370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.487233    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:48.487370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.487246    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:48.487370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.487258    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:48.489183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.489154    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4761326c-38e8-466e-8946-3c75605d0ad7-istio-podinfo" pod "4761326c-38e8-466e-8946-3c75605d0ad7" (UID: "4761326c-38e8-466e-8946-3c75605d0ad7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:32:48.490094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.490071    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-kube-api-access-mt2dn" pod "4761326c-38e8-466e-8946-3c75605d0ad7" (UID: "4761326c-38e8-466e-8946-3c75605d0ad7"). InnerVolumeSpecName "kube-api-access-mt2dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:48.494706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.494684    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-envoy" pod "4761326c-38e8-466e-8946-3c75605d0ad7" (UID: "4761326c-38e8-466e-8946-3c75605d0ad7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:32:48.496084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.496059    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-istio-token" pod "4761326c-38e8-466e-8946-3c75605d0ad7" (UID: "4761326c-38e8-466e-8946-3c75605d0ad7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:48.588786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.588756    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4761326c-38e8-466e-8946-3c75605d0ad7-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:48.588786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.588784    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4761326c-38e8-466e-8946-3c75605d0ad7-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:48.588786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.588793    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt2dn\" (UniqueName: \"kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-kube-api-access-mt2dn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:48.589035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:48.588803    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4761326c-38e8-466e-8946-3c75605d0ad7-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:49.375849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:49.375823    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"
Oct 29 22:32:49.449475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:49.449437    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"]
Oct 29 22:32:49.463233 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:49.463203    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-nqsf5"]
Oct 29 22:32:50.056941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:50.056907    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:50.118498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:50.118465    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4761326c-38e8-466e-8946-3c75605d0ad7" path="/var/lib/kubelet/pods/4761326c-38e8-466e-8946-3c75605d0ad7/volumes"
Oct 29 22:32:51.674955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:51.674915    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:52.388391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.388356    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9095-133e70"]
Oct 29 22:32:52.388949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.388935    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6266e819-5a4a-4dd2-a11c-b008142baf37" containerName="revoke-keys"
Oct 29 22:32:52.388993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.388950    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6266e819-5a4a-4dd2-a11c-b008142baf37" containerName="revoke-keys"
Oct 29 22:32:52.388993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.388966    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4761326c-38e8-466e-8946-3c75605d0ad7" containerName="istio-proxy"
Oct 29 22:32:52.388993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.388971    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4761326c-38e8-466e-8946-3c75605d0ad7" containerName="istio-proxy"
Oct 29 22:32:52.389100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.389089    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6266e819-5a4a-4dd2-a11c-b008142baf37" containerName="revoke-keys"
Oct 29 22:32:52.389132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.389102    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4761326c-38e8-466e-8946-3c75605d0ad7" containerName="istio-proxy"
Oct 29 22:32:52.394290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.394272    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9095-133e70"
Oct 29 22:32:52.407513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.407489    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9095-133e70"]
Oct 29 22:32:52.446613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.446586    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xl57t\""
Oct 29 22:32:52.539302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.539271    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgpxn\" (UniqueName: \"kubernetes.io/projected/a6b10e33-441e-41b3-a1e8-1ff386a105ed-kube-api-access-rgpxn\") pod \"test-curl-9095-133e70\" (UID: \"a6b10e33-441e-41b3-a1e8-1ff386a105ed\") " pod="opendatahub/test-curl-9095-133e70"
Oct 29 22:32:52.640554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.640480    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgpxn\" (UniqueName: \"kubernetes.io/projected/a6b10e33-441e-41b3-a1e8-1ff386a105ed-kube-api-access-rgpxn\") pod \"test-curl-9095-133e70\" (UID: \"a6b10e33-441e-41b3-a1e8-1ff386a105ed\") " pod="opendatahub/test-curl-9095-133e70"
Oct 29 22:32:52.661434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.661408    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgpxn\" (UniqueName: \"kubernetes.io/projected/a6b10e33-441e-41b3-a1e8-1ff386a105ed-kube-api-access-rgpxn\") pod \"test-curl-9095-133e70\" (UID: \"a6b10e33-441e-41b3-a1e8-1ff386a105ed\") " pod="opendatahub/test-curl-9095-133e70"
Oct 29 22:32:52.709713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:52.709689    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9095-133e70"
Oct 29 22:32:53.008506 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:53.008477    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6b10e33_441e_41b3_a1e8_1ff386a105ed.slice/crio-c00a6c5da7f7f2f9bab98688e8b74abd2f62b7805b93e4b6ed485514be80961f WatchSource:0}: Error finding container c00a6c5da7f7f2f9bab98688e8b74abd2f62b7805b93e4b6ed485514be80961f: Status 404 returned error can't find the container with id c00a6c5da7f7f2f9bab98688e8b74abd2f62b7805b93e4b6ed485514be80961f
Oct 29 22:32:53.014732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:53.014695    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9095-133e70"]
Oct 29 22:32:53.338001 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:53.337923    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:53.427417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:53.427381    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9095-133e70" event={"ID":"a6b10e33-441e-41b3-a1e8-1ff386a105ed","Type":"ContainerStarted","Data":"c00a6c5da7f7f2f9bab98688e8b74abd2f62b7805b93e4b6ed485514be80961f"}
Oct 29 22:32:54.033130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.033101    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"]
Oct 29 22:32:54.038280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.038253    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:54.058224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.057824    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"]
Oct 29 22:32:54.073496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.072617    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:32:54.073496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.072686    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:32:54.073496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.072739    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:32:54.073496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.072741    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:32:54.160227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.160189    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a6b90f5-8jcs2\" (UID: \"d114b79a-fab3-42f2-afe2-254d781491de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:54.160425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.160255    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92n2h\" (UniqueName: \"kubernetes.io/projected/d114b79a-fab3-42f2-afe2-254d781491de-kube-api-access-92n2h\") pod \"maas-api-revoke-sub-8a6b90f5-8jcs2\" (UID: \"d114b79a-fab3-42f2-afe2-254d781491de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:54.160425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.160403    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a6b90f5-8jcs2\" (UID: \"d114b79a-fab3-42f2-afe2-254d781491de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:54.261589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.261554    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a6b90f5-8jcs2\" (UID: \"d114b79a-fab3-42f2-afe2-254d781491de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:54.261770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.261617    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92n2h\" (UniqueName: \"kubernetes.io/projected/d114b79a-fab3-42f2-afe2-254d781491de-kube-api-access-92n2h\") pod \"maas-api-revoke-sub-8a6b90f5-8jcs2\" (UID: \"d114b79a-fab3-42f2-afe2-254d781491de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:54.261770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.261672    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a6b90f5-8jcs2\" (UID: \"d114b79a-fab3-42f2-afe2-254d781491de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:54.262471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.262426    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a6b90f5-8jcs2\" (UID: \"d114b79a-fab3-42f2-afe2-254d781491de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:54.264636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.264611    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a6b90f5-8jcs2\" (UID: \"d114b79a-fab3-42f2-afe2-254d781491de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:54.283024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.282997    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92n2h\" (UniqueName: \"kubernetes.io/projected/d114b79a-fab3-42f2-afe2-254d781491de-kube-api-access-92n2h\") pod \"maas-api-revoke-sub-8a6b90f5-8jcs2\" (UID: \"d114b79a-fab3-42f2-afe2-254d781491de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:54.362655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.362593    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:54.683522 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:54.683482    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd114b79a_fab3_42f2_afe2_254d781491de.slice/crio-476f14c38caa9876f727a3a597038b4e74742675cb64732b1b6d9437f52a949d WatchSource:0}: Error finding container 476f14c38caa9876f727a3a597038b4e74742675cb64732b1b6d9437f52a949d: Status 404 returned error can't find the container with id 476f14c38caa9876f727a3a597038b4e74742675cb64732b1b6d9437f52a949d
Oct 29 22:32:54.688605 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.688553    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"]
Oct 29 22:32:54.942307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:54.942274    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:55.454209 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:55.454177    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2" event={"ID":"d114b79a-fab3-42f2-afe2-254d781491de","Type":"ContainerStarted","Data":"510d62aaca6d91378c892bd5a58ace1bf9cdc9fd46149e5bded1285fbf821526"}
Oct 29 22:32:55.454651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:55.454238    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2" event={"ID":"d114b79a-fab3-42f2-afe2-254d781491de","Type":"ContainerStarted","Data":"476f14c38caa9876f727a3a597038b4e74742675cb64732b1b6d9437f52a949d"}
Oct 29 22:32:55.455863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:55.455837    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9095-133e70" event={"ID":"a6b10e33-441e-41b3-a1e8-1ff386a105ed","Type":"ContainerStarted","Data":"314864a4acb90171ad10e4a741ed6148e12d3dbc502ecba64b393f77e72f7de2"}
Oct 29 22:32:55.501075 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:55.501035    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2" podStartSLOduration=1.5010225579999998 podStartE2EDuration="1.501022558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:55.495806077 +0000 UTC m=+1117.520582209" watchObservedRunningTime="2026-10-29 22:32:55.501022558 +0000 UTC m=+1117.525798718"
Oct 29 22:32:55.536215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:55.536175    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9095-133e70" podStartSLOduration=1.45218375 podStartE2EDuration="3.53616319s" totalImagesPullingTime="2.08397944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:55.534267057 +0000 UTC m=+1117.559043204" watchObservedRunningTime="2026-10-29 22:32:55.53616319 +0000 UTC m=+1117.560939320"
Oct 29 22:32:56.121870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.121779    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:56.387037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.386961    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9095-133e70"]
Oct 29 22:32:56.394896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.394866    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9095-133e70"]
Oct 29 22:32:56.466245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.466213    4860 generic.go:356] "Generic (PLEG): container finished" podID="d114b79a-fab3-42f2-afe2-254d781491de" containerID="510d62aaca6d91378c892bd5a58ace1bf9cdc9fd46149e5bded1285fbf821526" exitCode=0
Oct 29 22:32:56.466665 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.466293    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2" event={"ID":"d114b79a-fab3-42f2-afe2-254d781491de","Type":"ContainerDied","Data":"510d62aaca6d91378c892bd5a58ace1bf9cdc9fd46149e5bded1285fbf821526"}
Oct 29 22:32:56.509767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.509738    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"]
Oct 29 22:32:56.510396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.510382    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a6b10e33-441e-41b3-a1e8-1ff386a105ed" containerName="test-curl-9095-133e70"
Oct 29 22:32:56.510441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.510397    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b10e33-441e-41b3-a1e8-1ff386a105ed" containerName="test-curl-9095-133e70"
Oct 29 22:32:56.510514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.510505    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b10e33-441e-41b3-a1e8-1ff386a105ed" containerName="test-curl-9095-133e70"
Oct 29 22:32:56.514529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.514508    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.532845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.532801    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f7dea32d-openshift-default-dockercfg-c9kk6\""
Oct 29 22:32:56.540003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.539967    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"]
Oct 29 22:32:56.575956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.575885    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:56.596189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.596156    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-data\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.596319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.596290    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b030dda-fbf0-4d08-800a-d8688aef34c6-istiod-ca-cert\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.596462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.596354    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shfzg\" (UniqueName: \"kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-kube-api-access-shfzg\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.596462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.596382    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-podinfo\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.596559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.596473    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-token\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.596648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.596628    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-certs\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.596702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.596683    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-credential-socket\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.596759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.596744    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-envoy\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.596817 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.596804    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-socket\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.697614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.697590    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-podinfo\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.697737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.697620    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-token\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.697737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.697680    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-certs\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.697737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.697711    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-credential-socket\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.697737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.697731    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-envoy\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.697950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.697786    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-socket\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.697950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.697834    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-data\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.697950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.697924    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b030dda-fbf0-4d08-800a-d8688aef34c6-istiod-ca-cert\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.698096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.697968    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shfzg\" (UniqueName: \"kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-kube-api-access-shfzg\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.698156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.698123    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-credential-socket\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.698211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.698163    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-certs\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.698353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.698313    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-socket\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.698611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.698587    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-data\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.698689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.698629    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b030dda-fbf0-4d08-800a-d8688aef34c6-istiod-ca-cert\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.700298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.700280    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-podinfo\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.700410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.700388    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-envoy\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.723893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.723870    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:56.728352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.728310    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-token\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.728847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.728831    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shfzg\" (UniqueName: \"kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-kube-api-access-shfzg\") pod \"e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") " pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.840372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.840304    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:56.862050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:56.862022    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:57.167774 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:32:57.167748    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b030dda_fbf0_4d08_800a_d8688aef34c6.slice/crio-d7a7cadefc528d2314a4ebe9b79b95698bfc14d33524ebd154afe0320ac19baa WatchSource:0}: Error finding container d7a7cadefc528d2314a4ebe9b79b95698bfc14d33524ebd154afe0320ac19baa: Status 404 returned error can't find the container with id d7a7cadefc528d2314a4ebe9b79b95698bfc14d33524ebd154afe0320ac19baa
Oct 29 22:32:57.170335 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.170286    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:32:57.170430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.170372    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:32:57.170430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.170404    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:32:57.172017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.171991    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"]
Oct 29 22:32:57.244924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.244897    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:57.375077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.375039    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:57.477627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.477590    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" event={"ID":"8b030dda-fbf0-4d08-800a-d8688aef34c6","Type":"ContainerStarted","Data":"423001e95419c4a4e48de09fd56b19e65c3941204e573cab6a632b0504f6a3ec"}
Oct 29 22:32:57.478132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.477654    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" event={"ID":"8b030dda-fbf0-4d08-800a-d8688aef34c6","Type":"ContainerStarted","Data":"d7a7cadefc528d2314a4ebe9b79b95698bfc14d33524ebd154afe0320ac19baa"}
Oct 29 22:32:57.478132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.477765    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9095-133e70" podUID="a6b10e33-441e-41b3-a1e8-1ff386a105ed" containerName="test-curl-9095-133e70" containerID="cri-o://314864a4acb90171ad10e4a741ed6148e12d3dbc502ecba64b393f77e72f7de2" gracePeriod=2
Oct 29 22:32:57.493068 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:32:57.493030    4860 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9095-133e70\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" podUID="a6b10e33-441e-41b3-a1e8-1ff386a105ed" pod="opendatahub/test-curl-9095-133e70"
Oct 29 22:32:57.547531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.547477    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" podStartSLOduration=1.547462916 podStartE2EDuration="1.547462916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:57.542054237 +0000 UTC m=+1119.566830396" watchObservedRunningTime="2026-10-29 22:32:57.547462916 +0000 UTC m=+1119.572239047"
Oct 29 22:32:57.564764 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:32:57.564730    4860 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9095-133e70\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" podUID="a6b10e33-441e-41b3-a1e8-1ff386a105ed" pod="opendatahub/test-curl-9095-133e70"
Oct 29 22:32:57.627943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.627917    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:57.711815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.711780    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-xks-ca\") pod \"d114b79a-fab3-42f2-afe2-254d781491de\" (UID: \"d114b79a-fab3-42f2-afe2-254d781491de\") "
Oct 29 22:32:57.711964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.711893    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-openshift-ca\") pod \"d114b79a-fab3-42f2-afe2-254d781491de\" (UID: \"d114b79a-fab3-42f2-afe2-254d781491de\") "
Oct 29 22:32:57.712055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.711983    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d114b79a-fab3-42f2-afe2-254d781491de-kube-api-access-92n2h\" (UniqueName: \"kubernetes.io/projected/d114b79a-fab3-42f2-afe2-254d781491de-kube-api-access-92n2h\") pod \"d114b79a-fab3-42f2-afe2-254d781491de\" (UID: \"d114b79a-fab3-42f2-afe2-254d781491de\") "
Oct 29 22:32:57.712525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.712490    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-openshift-ca" pod "d114b79a-fab3-42f2-afe2-254d781491de" (UID: "d114b79a-fab3-42f2-afe2-254d781491de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:32:57.715121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.715079    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d114b79a-fab3-42f2-afe2-254d781491de-kube-api-access-92n2h" pod "d114b79a-fab3-42f2-afe2-254d781491de" (UID: "d114b79a-fab3-42f2-afe2-254d781491de"). InnerVolumeSpecName "kube-api-access-92n2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:32:57.715470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.715428    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-xks-ca" pod "d114b79a-fab3-42f2-afe2-254d781491de" (UID: "d114b79a-fab3-42f2-afe2-254d781491de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:32:57.717260 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:32:57.717225    4860 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9095-133e70\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" podUID="a6b10e33-441e-41b3-a1e8-1ff386a105ed" pod="opendatahub/test-curl-9095-133e70"
Oct 29 22:32:57.816346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.814814    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:57.816346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.814851    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92n2h\" (UniqueName: \"kubernetes.io/projected/d114b79a-fab3-42f2-afe2-254d781491de-kube-api-access-92n2h\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:57.816346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.814867    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d114b79a-fab3-42f2-afe2-254d781491de-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:32:57.832372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.824448    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:57.841241 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:57.840428    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:58.126840 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:32:58.126805    4860 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9095-133e70\" is forbidden: User \"system:node:ip-10-0-128-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-31.ec2.internal' and this object" podUID="a6b10e33-441e-41b3-a1e8-1ff386a105ed" pod="opendatahub/test-curl-9095-133e70"
Oct 29 22:32:58.316564 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:58.316530    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:58.487648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:58.487624    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"
Oct 29 22:32:58.488033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:58.487666    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2" event={"ID":"d114b79a-fab3-42f2-afe2-254d781491de","Type":"ContainerDied","Data":"476f14c38caa9876f727a3a597038b4e74742675cb64732b1b6d9437f52a949d"}
Oct 29 22:32:58.488033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:58.487714    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="476f14c38caa9876f727a3a597038b4e74742675cb64732b1b6d9437f52a949d"
Oct 29 22:32:58.693344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:58.692951    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:58.846802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:58.846732    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:59.272078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:59.271950    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:59.424452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:59.424411    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:32:59.499167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:59.499138    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:59.500278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:59.500255    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:32:59.916554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:59.916530    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9095-133e70"
Oct 29 22:32:59.974930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:59.974877    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b10e33-441e-41b3-a1e8-1ff386a105ed-kube-api-access-rgpxn\" (UniqueName: \"kubernetes.io/projected/a6b10e33-441e-41b3-a1e8-1ff386a105ed-kube-api-access-rgpxn\") pod \"a6b10e33-441e-41b3-a1e8-1ff386a105ed\" (UID: \"a6b10e33-441e-41b3-a1e8-1ff386a105ed\") "
Oct 29 22:32:59.978894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:32:59.978846    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b10e33-441e-41b3-a1e8-1ff386a105ed-kube-api-access-rgpxn" pod "a6b10e33-441e-41b3-a1e8-1ff386a105ed" (UID: "a6b10e33-441e-41b3-a1e8-1ff386a105ed"). InnerVolumeSpecName "kube-api-access-rgpxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:33:00.076742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:00.076697    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgpxn\" (UniqueName: \"kubernetes.io/projected/a6b10e33-441e-41b3-a1e8-1ff386a105ed-kube-api-access-rgpxn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:00.121559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:00.120948    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b10e33-441e-41b3-a1e8-1ff386a105ed" path="/var/lib/kubelet/pods/a6b10e33-441e-41b3-a1e8-1ff386a105ed/volumes"
Oct 29 22:33:00.558368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:00.553951    4860 generic.go:356] "Generic (PLEG): container finished" podID="a6b10e33-441e-41b3-a1e8-1ff386a105ed" containerID="314864a4acb90171ad10e4a741ed6148e12d3dbc502ecba64b393f77e72f7de2" exitCode=137
Oct 29 22:33:00.558368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:00.555135    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9095-133e70"
Oct 29 22:33:00.558368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:00.555404    4860 scope.go:122] "RemoveContainer" containerID="314864a4acb90171ad10e4a741ed6148e12d3dbc502ecba64b393f77e72f7de2"
Oct 29 22:33:00.623464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:00.623438    4860 scope.go:122] "RemoveContainer" containerID="314864a4acb90171ad10e4a741ed6148e12d3dbc502ecba64b393f77e72f7de2"
Oct 29 22:33:00.626111 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:33:00.626033    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314864a4acb90171ad10e4a741ed6148e12d3dbc502ecba64b393f77e72f7de2\": container with ID starting with 314864a4acb90171ad10e4a741ed6148e12d3dbc502ecba64b393f77e72f7de2 not found: ID does not exist" containerID="314864a4acb90171ad10e4a741ed6148e12d3dbc502ecba64b393f77e72f7de2"
Oct 29 22:33:00.626111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:00.626072    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314864a4acb90171ad10e4a741ed6148e12d3dbc502ecba64b393f77e72f7de2"} err="failed to get container status \"314864a4acb90171ad10e4a741ed6148e12d3dbc502ecba64b393f77e72f7de2\": rpc error: code = NotFound desc = could not find container \"314864a4acb90171ad10e4a741ed6148e12d3dbc502ecba64b393f77e72f7de2\": container with ID starting with 314864a4acb90171ad10e4a741ed6148e12d3dbc502ecba64b393f77e72f7de2 not found: ID does not exist"
Oct 29 22:33:01.297787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:01.297749    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:01.360496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:01.360452    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:01.362062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:01.362032    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:01.366459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:01.362238    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Oct 29 22:33:01.393579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:01.393531    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:01.440181 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:01.440096    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:01.524962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:01.524921    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:01.651406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:01.650424    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:01.701351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:01.671421    4860 patch_prober.go:29] interesting pod/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:33:01.701351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:01.671490    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.019318 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.019267    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:02.170365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.035993    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"]
Oct 29 22:33:02.170365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.097388    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d114b79a-fab3-42f2-afe2-254d781491de" containerName="revoke-keys"
Oct 29 22:33:02.170365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.097412    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d114b79a-fab3-42f2-afe2-254d781491de" containerName="revoke-keys"
Oct 29 22:33:02.170365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.097651    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d114b79a-fab3-42f2-afe2-254d781491de" containerName="revoke-keys"
Oct 29 22:33:02.246362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.192464    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.246362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.199787    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:33:02.246362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.199972    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:33:02.246362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.205672    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.275407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.260284    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.387423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.372510    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:02.387423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.372652    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:33:02.387423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.372900    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.418628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.388990    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"
Oct 29 22:33:02.421023 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.420983    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.615371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.439171    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f7dea32d\""
Oct 29 22:33:02.620998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.488525    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.843644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.519678    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f7dea32d-dockercfg-wg7l5\""
Oct 29 22:33:02.855869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.545059    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04d68764-3b0d-48c9-ad1e-6a8078516fe3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr\" (UID: \"04d68764-3b0d-48c9-ad1e-6a8078516fe3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"
Oct 29 22:33:02.856865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.856832    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpdsz\" (UniqueName: \"kubernetes.io/projected/04d68764-3b0d-48c9-ad1e-6a8078516fe3-kube-api-access-wpdsz\") pod \"payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr\" (UID: \"04d68764-3b0d-48c9-ad1e-6a8078516fe3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"
Oct 29 22:33:02.861855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.570653    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:33:02.862293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.862277    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:33:02.862619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.862606    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:33:02.862784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.862774    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:33:02.862880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.862870    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"]
Oct 29 22:33:02.863025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.863014    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"]
Oct 29 22:33:02.865161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.659014    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.866448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.583381    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:02.869888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.736040    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:33:02.926108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.926075    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:33:02.926362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.926347    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:33:02.926504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.926493    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"]
Oct 29 22:33:02.926627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.926617    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:33:02.926779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.926768    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"
Oct 29 22:33:02.940690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.940603    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:33:02.954781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.954760    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:33:02.963499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.963377    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpdsz\" (UniqueName: \"kubernetes.io/projected/04d68764-3b0d-48c9-ad1e-6a8078516fe3-kube-api-access-wpdsz\") pod \"payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr\" (UID: \"04d68764-3b0d-48c9-ad1e-6a8078516fe3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"
Oct 29 22:33:02.963632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.963585    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04d68764-3b0d-48c9-ad1e-6a8078516fe3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr\" (UID: \"04d68764-3b0d-48c9-ad1e-6a8078516fe3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"
Oct 29 22:33:02.965184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.965162    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04d68764-3b0d-48c9-ad1e-6a8078516fe3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr\" (UID: \"04d68764-3b0d-48c9-ad1e-6a8078516fe3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"
Oct 29 22:33:02.978007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.977988    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:33:02.989226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:02.989197    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpdsz\" (UniqueName: \"kubernetes.io/projected/04d68764-3b0d-48c9-ad1e-6a8078516fe3-kube-api-access-wpdsz\") pod \"payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr\" (UID: \"04d68764-3b0d-48c9-ad1e-6a8078516fe3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"
Oct 29 22:33:03.003682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:03.003657    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:03.010341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:03.010144    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"
Oct 29 22:33:03.070377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:03.067053    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwwzh\" (UniqueName: \"kubernetes.io/projected/8ef676f8-6c61-44d2-b71f-85a401a4f984-kube-api-access-qwwzh\") pod \"payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx\" (UID: \"8ef676f8-6c61-44d2-b71f-85a401a4f984\") " pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"
Oct 29 22:33:03.070377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:03.067174    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef676f8-6c61-44d2-b71f-85a401a4f984-plugins-config-volume\") pod \"payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx\" (UID: \"8ef676f8-6c61-44d2-b71f-85a401a4f984\") " pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"
Oct 29 22:33:03.168957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:03.168849    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwwzh\" (UniqueName: \"kubernetes.io/projected/8ef676f8-6c61-44d2-b71f-85a401a4f984-kube-api-access-qwwzh\") pod \"payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx\" (UID: \"8ef676f8-6c61-44d2-b71f-85a401a4f984\") " pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"
Oct 29 22:33:03.169133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:03.168967    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef676f8-6c61-44d2-b71f-85a401a4f984-plugins-config-volume\") pod \"payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx\" (UID: \"8ef676f8-6c61-44d2-b71f-85a401a4f984\") " pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"
Oct 29 22:33:03.170645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:03.170619    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef676f8-6c61-44d2-b71f-85a401a4f984-plugins-config-volume\") pod \"payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx\" (UID: \"8ef676f8-6c61-44d2-b71f-85a401a4f984\") " pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"
Oct 29 22:33:03.190667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:03.190633    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwwzh\" (UniqueName: \"kubernetes.io/projected/8ef676f8-6c61-44d2-b71f-85a401a4f984-kube-api-access-qwwzh\") pod \"payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx\" (UID: \"8ef676f8-6c61-44d2-b71f-85a401a4f984\") " pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"
Oct 29 22:33:03.277315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:03.277285    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"
Oct 29 22:33:03.350705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:03.350670    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"]
Oct 29 22:33:03.596866 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:33:03.596837    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef676f8_6c61_44d2_b71f_85a401a4f984.slice/crio-10a86e249d4da77e5e2b9a1e9e8eece7ba22271852057f0decd58b1a5496d4e8 WatchSource:0}: Error finding container 10a86e249d4da77e5e2b9a1e9e8eece7ba22271852057f0decd58b1a5496d4e8: Status 404 returned error can't find the container with id 10a86e249d4da77e5e2b9a1e9e8eece7ba22271852057f0decd58b1a5496d4e8
Oct 29 22:33:03.601557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:03.601527    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"]
Oct 29 22:33:03.974067 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:03.974031    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:04.239542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.239439    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx" event={"ID":"8ef676f8-6c61-44d2-b71f-85a401a4f984","Type":"ContainerStarted","Data":"19a9042e8c22cebf9bcab4d820106cfcd99b73e36cfd6f9b56215f3d754643bf"}
Oct 29 22:33:04.241451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.241421    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx" event={"ID":"8ef676f8-6c61-44d2-b71f-85a401a4f984","Type":"ContainerStarted","Data":"10a86e249d4da77e5e2b9a1e9e8eece7ba22271852057f0decd58b1a5496d4e8"}
Oct 29 22:33:04.241695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.241681    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"
Oct 29 22:33:04.244583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.244525    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr" event={"ID":"04d68764-3b0d-48c9-ad1e-6a8078516fe3","Type":"ContainerStarted","Data":"962c1ec652ba591c4c5eda74466a2d6b3a8d0353c8484ff0b9cd15d931170895"}
Oct 29 22:33:04.244739 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.244699    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr" event={"ID":"04d68764-3b0d-48c9-ad1e-6a8078516fe3","Type":"ContainerStarted","Data":"06e33cfb6b3a6241c61f4bc6e2d3fc4a8d32625d1ca1f96ae39d76ea5b598699"}
Oct 29 22:33:04.246206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.246101    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"
Oct 29 22:33:04.293309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.293260    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx" podStartSLOduration=2.293246526 podStartE2EDuration="2.293246526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:04.288544907 +0000 UTC m=+1126.313321039" watchObservedRunningTime="2026-10-29 22:33:04.293246526 +0000 UTC m=+1126.318022657"
Oct 29 22:33:04.315755 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.315725    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:04.340433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.340369    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr" podStartSLOduration=3.340348546 podStartE2EDuration="3.340348546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:04.336788463 +0000 UTC m=+1126.361564623" watchObservedRunningTime="2026-10-29 22:33:04.340348546 +0000 UTC m=+1126.365124680"
Oct 29 22:33:04.454641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.454606    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:04.478953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.478910    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"]
Oct 29 22:33:04.483480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.483442    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"
Oct 29 22:33:04.503766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.503737    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"]
Oct 29 22:33:04.505507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.505486    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dc369ed0-dockercfg-mszw4\""
Oct 29 22:33:04.505507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.505486    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dc369ed0\""
Oct 29 22:33:04.589309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.589278    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86650176-e767-4503-8a16-43c9761da511-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch\" (UID: \"86650176-e767-4503-8a16-43c9761da511\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"
Oct 29 22:33:04.589511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.589427    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnclh\" (UniqueName: \"kubernetes.io/projected/86650176-e767-4503-8a16-43c9761da511-kube-api-access-fnclh\") pod \"payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch\" (UID: \"86650176-e767-4503-8a16-43c9761da511\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"
Oct 29 22:33:04.713112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.712518    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86650176-e767-4503-8a16-43c9761da511-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch\" (UID: \"86650176-e767-4503-8a16-43c9761da511\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"
Oct 29 22:33:04.713112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.712612    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnclh\" (UniqueName: \"kubernetes.io/projected/86650176-e767-4503-8a16-43c9761da511-kube-api-access-fnclh\") pod \"payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch\" (UID: \"86650176-e767-4503-8a16-43c9761da511\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"
Oct 29 22:33:04.714282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.714236    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86650176-e767-4503-8a16-43c9761da511-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch\" (UID: \"86650176-e767-4503-8a16-43c9761da511\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"
Oct 29 22:33:04.735923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.735893    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnclh\" (UniqueName: \"kubernetes.io/projected/86650176-e767-4503-8a16-43c9761da511-kube-api-access-fnclh\") pod \"payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch\" (UID: \"86650176-e767-4503-8a16-43c9761da511\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"
Oct 29 22:33:04.803530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:04.803428    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"
Oct 29 22:33:05.028584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.028544    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:05.141817 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:33:05.141776    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86650176_e767_4503_8a16_43c9761da511.slice/crio-fc4aa22737e0f516f64ba07a999e6e830b6df4aaee27095a2a1c9f4b0fcdfc3d WatchSource:0}: Error finding container fc4aa22737e0f516f64ba07a999e6e830b6df4aaee27095a2a1c9f4b0fcdfc3d: Status 404 returned error can't find the container with id fc4aa22737e0f516f64ba07a999e6e830b6df4aaee27095a2a1c9f4b0fcdfc3d
Oct 29 22:33:05.148821 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.148794    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"]
Oct 29 22:33:05.240529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.240501    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:05.255905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.255881    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch" event={"ID":"86650176-e767-4503-8a16-43c9761da511","Type":"ContainerStarted","Data":"b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5"}
Oct 29 22:33:05.255997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.255928    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch" event={"ID":"86650176-e767-4503-8a16-43c9761da511","Type":"ContainerStarted","Data":"fc4aa22737e0f516f64ba07a999e6e830b6df4aaee27095a2a1c9f4b0fcdfc3d"}
Oct 29 22:33:05.255997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.255959    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"
Oct 29 22:33:05.324969 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.324932    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch" podStartSLOduration=1.324919373 podStartE2EDuration="1.324919373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:05.319108015 +0000 UTC m=+1127.343884146" watchObservedRunningTime="2026-10-29 22:33:05.324919373 +0000 UTC m=+1127.349695540"
Oct 29 22:33:05.615894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.615858    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"]
Oct 29 22:33:05.620855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.620835    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.647577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.647554    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"]
Oct 29 22:33:05.662473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.662449    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:05.731632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.731588    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.731837 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.731646    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.731837 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.731729    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.731837 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.731762    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88686c92-66f4-4846-8c9e-bf969f13efd4-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.731837 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.731789    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.732083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.731927    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxtm\" (UniqueName: \"kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-kube-api-access-qdxtm\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.732083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.731971    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.732083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.732001    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.732083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.732044    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.792742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.792713    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:05.833507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.833473    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.833687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.833540    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.833687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.833577    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.833687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.833617    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88686c92-66f4-4846-8c9e-bf969f13efd4-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.833687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.833657    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.833906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.833731    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxtm\" (UniqueName: \"kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-kube-api-access-qdxtm\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.833906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.833772    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.833906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.833808    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.833906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.833868    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.834088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.833957    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.834088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.834002    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.834231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.834196    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.834294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.834250    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.834434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.834409    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88686c92-66f4-4846-8c9e-bf969f13efd4-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.836134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.836110    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.836218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.836172    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.855301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.855271    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.855446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.855430    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxtm\" (UniqueName: \"kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-kube-api-access-qdxtm\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:05.916574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.916500    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:05.935228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:05.935205    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:06.116853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.116823    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"]
Oct 29 22:33:06.120652 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.120624    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.145049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.145022    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"]
Oct 29 22:33:06.251630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.251601    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.251781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.251654    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.251781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.251740    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.251884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.251785    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.251884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.251817    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.251955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.251921    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.252002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.251956    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvkd\" (UniqueName: \"kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-kube-api-access-dlvkd\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.252052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.252039    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.252092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.252080    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.285515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.285482    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"]
Oct 29 22:33:06.292310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.292272    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:33:06.292446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.292365    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:33:06.292446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.292413    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:33:06.352885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.352853    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.352996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.352914    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvkd\" (UniqueName: \"kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-kube-api-access-dlvkd\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.352996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.352958    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.353115 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.353000    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.353176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.353146    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.353229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.353187    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.353229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.353205    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.353342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.353259    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.353342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.353287    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.354378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.353837    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-certs\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.354378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.353877    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.354378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.354014    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-data\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.354378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.354287    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istiod-ca-cert\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.354690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.354618    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-credential-socket\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.356425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.356402    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-envoy\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.356572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.356555    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-podinfo\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.379528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.379315    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-token\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.379528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.379443    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvkd\" (UniqueName: \"kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-kube-api-access-dlvkd\") pod \"e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") " pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.404991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.404958    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:06.435411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.435378    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:06.779698 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:33:06.779661    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2aad81_e61d_4edb_85c8_0c8e725d00a5.slice/crio-2a2cb279d631899420c12df4b9a3c43d268df45a49be2f53c101b15cf692edc1 WatchSource:0}: Error finding container 2a2cb279d631899420c12df4b9a3c43d268df45a49be2f53c101b15cf692edc1: Status 404 returned error can't find the container with id 2a2cb279d631899420c12df4b9a3c43d268df45a49be2f53c101b15cf692edc1
Oct 29 22:33:06.782365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.782311    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:33:06.782482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.782414    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:33:06.782482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.782457    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:33:06.783028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.782960    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"]
Oct 29 22:33:06.904002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:06.903941    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:07.026415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.023300    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:07.304199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.304103    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" event={"ID":"4c2aad81-e61d-4edb-85c8-0c8e725d00a5","Type":"ContainerStarted","Data":"6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125"}
Oct 29 22:33:07.304199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.304161    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" event={"ID":"4c2aad81-e61d-4edb-85c8-0c8e725d00a5","Type":"ContainerStarted","Data":"2a2cb279d631899420c12df4b9a3c43d268df45a49be2f53c101b15cf692edc1"}
Oct 29 22:33:07.305943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.305917    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4" event={"ID":"88686c92-66f4-4846-8c9e-bf969f13efd4","Type":"ContainerStarted","Data":"f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79"}
Oct 29 22:33:07.306090 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.305968    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4" event={"ID":"88686c92-66f4-4846-8c9e-bf969f13efd4","Type":"ContainerStarted","Data":"f390eeb3c49a826eb3f63ac4a5d1a2e4aed0dfa2bfce233bba90f3ea021bec66"}
Oct 29 22:33:07.398609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.398560    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" podStartSLOduration=1.398546027 podStartE2EDuration="1.398546027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:07.366171888 +0000 UTC m=+1129.390948019" watchObservedRunningTime="2026-10-29 22:33:07.398546027 +0000 UTC m=+1129.423322157"
Oct 29 22:33:07.419710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.419660    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4" podStartSLOduration=2.4196469 podStartE2EDuration="2.4196469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:07.417219438 +0000 UTC m=+1129.441995595" watchObservedRunningTime="2026-10-29 22:33:07.4196469 +0000 UTC m=+1129.444423030"
Oct 29 22:33:07.436479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.436453    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:07.437601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.437577    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:33:07.437682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.437621    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" podUID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 29 22:33:07.518422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.518380    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"]
Oct 29 22:33:07.522425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.522402    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:07.538376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.538351    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:33:07.538524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.538461    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:33:07.538524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.538511    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:33:07.538524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.538522    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:33:07.540710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.540692    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"]
Oct 29 22:33:07.595079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.594991    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:07.701353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.697282    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555e24ad-n5cfz\" (UID: \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:07.701353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.697491    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvslz\" (UniqueName: \"kubernetes.io/projected/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-kube-api-access-qvslz\") pod \"maas-api-revoke-sub-555e24ad-n5cfz\" (UID: \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:07.701353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.697533    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555e24ad-n5cfz\" (UID: \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:07.801352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.800294    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvslz\" (UniqueName: \"kubernetes.io/projected/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-kube-api-access-qvslz\") pod \"maas-api-revoke-sub-555e24ad-n5cfz\" (UID: \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:07.801352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.800383    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555e24ad-n5cfz\" (UID: \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:07.801352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.800589    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555e24ad-n5cfz\" (UID: \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:07.801667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.801613    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555e24ad-n5cfz\" (UID: \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:07.822591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.822554    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555e24ad-n5cfz\" (UID: \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:07.823475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.823450    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvslz\" (UniqueName: \"kubernetes.io/projected/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-kube-api-access-qvslz\") pod \"maas-api-revoke-sub-555e24ad-n5cfz\" (UID: \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:07.837487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.837466    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:07.935648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.935614    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:07.937367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.937337    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 29 22:33:07.937502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:07.937392    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4" podUID="88686c92-66f4-4846-8c9e-bf969f13efd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 29 22:33:08.232234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:08.232016    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"]
Oct 29 22:33:08.239661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:08.239596    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:08.332840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:08.332805    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz" event={"ID":"c7bb2813-2dbf-4245-92cd-a1278a0f48a7","Type":"ContainerStarted","Data":"223550b0941f747e0e340cc9c2c38ad28bfd12675d150d29c4b6b68d63bcdab6"}
Oct 29 22:33:08.465345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:08.462720    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:33:08.465345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:08.462782    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" podUID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 29 22:33:08.936259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:08.936219    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 29 22:33:08.936453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:08.936278    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4" podUID="88686c92-66f4-4846-8c9e-bf969f13efd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 29 22:33:09.344138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:09.344059    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz" event={"ID":"c7bb2813-2dbf-4245-92cd-a1278a0f48a7","Type":"ContainerStarted","Data":"ea369d1ba889635e8625129760a7ba1a10a6d53d9ab447c8a87071f4fa454653"}
Oct 29 22:33:09.419288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:09.405390    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz" podStartSLOduration=2.4053671039999998 podStartE2EDuration="2.405367104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:09.390058433 +0000 UTC m=+1131.414834589" watchObservedRunningTime="2026-10-29 22:33:09.405367104 +0000 UTC m=+1131.430143263"
Oct 29 22:33:09.435861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:09.435836    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:33:09.436012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:09.435889    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" podUID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 29 22:33:09.521267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:09.521234    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:09.661421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:09.661311    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:09.816223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:09.816188    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:09.936832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:09.936384    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 29 22:33:09.936832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:09.936442    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4" podUID="88686c92-66f4-4846-8c9e-bf969f13efd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 29 22:33:10.355190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:10.355100    4860 generic.go:356] "Generic (PLEG): container finished" podID="c7bb2813-2dbf-4245-92cd-a1278a0f48a7" containerID="ea369d1ba889635e8625129760a7ba1a10a6d53d9ab447c8a87071f4fa454653" exitCode=0
Oct 29 22:33:10.355190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:10.355170    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz" event={"ID":"c7bb2813-2dbf-4245-92cd-a1278a0f48a7","Type":"ContainerDied","Data":"ea369d1ba889635e8625129760a7ba1a10a6d53d9ab447c8a87071f4fa454653"}
Oct 29 22:33:10.436520 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:10.436480    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:33:10.436691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:10.436532    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" podUID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 29 22:33:10.936348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:10.935635    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 29 22:33:10.936348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:10.935689    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4" podUID="88686c92-66f4-4846-8c9e-bf969f13efd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 29 22:33:11.149804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.149763    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:11.256623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.256535    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:11.436567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.436274    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:33:11.436567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.436356    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" podUID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 29 22:33:11.662256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.662231    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:11.664129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.664110    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:11.838957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.838908    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-kube-api-access-qvslz\" (UniqueName: \"kubernetes.io/projected/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-kube-api-access-qvslz\") pod \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\" (UID: \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\") "
Oct 29 22:33:11.839162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.839017    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-xks-ca\") pod \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\" (UID: \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\") "
Oct 29 22:33:11.839162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.839129    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-openshift-ca\") pod \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\" (UID: \"c7bb2813-2dbf-4245-92cd-a1278a0f48a7\") "
Oct 29 22:33:11.840051 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.840022    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-openshift-ca" pod "c7bb2813-2dbf-4245-92cd-a1278a0f48a7" (UID: "c7bb2813-2dbf-4245-92cd-a1278a0f48a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:33:11.842234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.842205    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-kube-api-access-qvslz" pod "c7bb2813-2dbf-4245-92cd-a1278a0f48a7" (UID: "c7bb2813-2dbf-4245-92cd-a1278a0f48a7"). InnerVolumeSpecName "kube-api-access-qvslz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:33:11.847706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.847678    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-xks-ca" pod "c7bb2813-2dbf-4245-92cd-a1278a0f48a7" (UID: "c7bb2813-2dbf-4245-92cd-a1278a0f48a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:33:11.936510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.936472    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 29 22:33:11.936657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.936534    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4" podUID="88686c92-66f4-4846-8c9e-bf969f13efd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 29 22:33:11.940800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.940778    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvslz\" (UniqueName: \"kubernetes.io/projected/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-kube-api-access-qvslz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:11.940800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.940802    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:11.940937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:11.940811    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bb2813-2dbf-4245-92cd-a1278a0f48a7-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:12.428037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:12.427955    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"
Oct 29 22:33:12.428037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:12.427996    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz" event={"ID":"c7bb2813-2dbf-4245-92cd-a1278a0f48a7","Type":"ContainerDied","Data":"223550b0941f747e0e340cc9c2c38ad28bfd12675d150d29c4b6b68d63bcdab6"}
Oct 29 22:33:12.428214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:12.428048    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="223550b0941f747e0e340cc9c2c38ad28bfd12675d150d29c4b6b68d63bcdab6"
Oct 29 22:33:12.436581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:12.436556    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:33:12.436936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:12.436598    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" podUID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 29 22:33:12.935837 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:12.935799    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 29 22:33:12.936006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:12.935865    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4" podUID="88686c92-66f4-4846-8c9e-bf969f13efd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 29 22:33:13.407693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:13.407574    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:13.436617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:13.436585    4860 patch_prober.go:29] interesting pod/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 29 22:33:13.437014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:13.436647    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" podUID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 29 22:33:13.939970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:13.939935    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:13.940453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:13.940432    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:13.941088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:13.941066    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:33:14.440666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:14.440633    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:14.441145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:14.441127    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:14.441789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:14.441769    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:33:15.136505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:15.136476    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:15.258586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:15.258550    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:15.259516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:15.259499    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"
Oct 29 22:33:15.383047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:15.382980    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:16.304616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:16.304577    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"
Oct 29 22:33:16.305731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:16.305693    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"
Oct 29 22:33:16.863790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:16.863756    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:17.928128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:17.928073    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:18.058335 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:33:18.058289    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:33:18.346482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:18.346454    4860 scope.go:122] "RemoveContainer" containerID="956f3c1df683e571a0088062426798ed37c6fe83c31684139b60188912b5cfbb"
Oct 29 22:33:18.495828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:18.495788    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:18.621357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:18.621255    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:18.732446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:18.732408    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:20.137095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:20.137063    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:21.694678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:21.694639    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:23.267355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:23.267306    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:23.414187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:23.414159    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:24.013376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:24.013342    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:25.006560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:25.006531    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:26.768349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:26.768243    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:28.804350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:28.803643    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:29.207126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:29.207091    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:30.478370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:30.478336    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:32.274623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:32.274588    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:33.040473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:33.040440    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"]
Oct 29 22:33:33.054478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:33.054451    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cffa770-4cv98"]
Oct 29 22:33:33.888434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:33.888361    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:34.114373 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:34.114340    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="215017aa-3527-489e-9ca4-aa473df6746a" path="/var/lib/kubelet/pods/215017aa-3527-489e-9ca4-aa473df6746a/volumes"
Oct 29 22:33:35.373550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.373511    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"]
Oct 29 22:33:35.374177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.374163    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c7bb2813-2dbf-4245-92cd-a1278a0f48a7" containerName="revoke-keys"
Oct 29 22:33:35.374221 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.374178    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7bb2813-2dbf-4245-92cd-a1278a0f48a7" containerName="revoke-keys"
Oct 29 22:33:35.374348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.374337    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7bb2813-2dbf-4245-92cd-a1278a0f48a7" containerName="revoke-keys"
Oct 29 22:33:35.381898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.381879    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:35.396618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.396588    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"]
Oct 29 22:33:35.401110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.401085    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:33:35.401263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.401117    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:33:35.401396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.401161    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:33:35.412004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.411983    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:33:35.428385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.428354    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt5pc\" (UniqueName: \"kubernetes.io/projected/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-kube-api-access-pt5pc\") pod \"maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v\" (UID: \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:35.428572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.428559    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v\" (UID: \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:35.428619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.428587    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v\" (UID: \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:35.530168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.530141    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt5pc\" (UniqueName: \"kubernetes.io/projected/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-kube-api-access-pt5pc\") pod \"maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v\" (UID: \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:35.530297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.530275    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v\" (UID: \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:35.530385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.530297    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v\" (UID: \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:35.530923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.530901    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v\" (UID: \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:35.532675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.532654    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v\" (UID: \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:35.550663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.550638    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt5pc\" (UniqueName: \"kubernetes.io/projected/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-kube-api-access-pt5pc\") pod \"maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v\" (UID: \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:35.606661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.606634    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"]
Oct 29 22:33:35.611486 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.611470    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.631185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.631116    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.631185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.631155    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.631507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.631192    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.631507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.631308    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.631507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.631355    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.631507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.631476    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.631746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.631538    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28324a2-e722-4488-81b2-dc8152adaf72-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.631746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.631589    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvqfs\" (UniqueName: \"kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-kube-api-access-pvqfs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.631746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.631651    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28324a2-e722-4488-81b2-dc8152adaf72-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.637016 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.636993    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"]
Oct 29 22:33:35.701563 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.701530    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:35.732916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.732886    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvqfs\" (UniqueName: \"kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-kube-api-access-pvqfs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.733011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.732955    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28324a2-e722-4488-81b2-dc8152adaf72-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.733052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.733041    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.733089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.733067    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.733124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.733087    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.733124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.733113    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.733192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.733136    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.733226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.733190    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.733266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.733231    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28324a2-e722-4488-81b2-dc8152adaf72-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.734058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.734032    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.734335 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.734305    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.734634 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.734595    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.734634 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.734624    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28324a2-e722-4488-81b2-dc8152adaf72-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.734875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.734847    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.736389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.736365    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28324a2-e722-4488-81b2-dc8152adaf72-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.736505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.736483    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.754399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.754372    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvqfs\" (UniqueName: \"kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-kube-api-access-pvqfs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.754748 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.754733    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:35.866673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.866634    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:35.926458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:35.926355    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:36.026776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:36.026641    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"]
Oct 29 22:33:36.277690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:36.277660    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"]
Oct 29 22:33:36.285266 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:33:36.285238    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28324a2_e722_4488_81b2_dc8152adaf72.slice/crio-43bda2fee5f9dd10e6c1cd2acca6569c4075c1d5a4010f4945aa7934dcfaac1f WatchSource:0}: Error finding container 43bda2fee5f9dd10e6c1cd2acca6569c4075c1d5a4010f4945aa7934dcfaac1f: Status 404 returned error can't find the container with id 43bda2fee5f9dd10e6c1cd2acca6569c4075c1d5a4010f4945aa7934dcfaac1f
Oct 29 22:33:36.288156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:36.288117    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:33:36.288239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:36.288201    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:33:36.288283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:36.288244    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:33:36.806108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:36.806067    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v" event={"ID":"41e1dcae-9c4c-442d-ad40-ec6df1cf620e","Type":"ContainerStarted","Data":"534e047eb87a69810c7fb09e1a0180b043c5a7abc5596e6e0e3afcd608352499"}
Oct 29 22:33:36.806615 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:36.806127    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v" event={"ID":"41e1dcae-9c4c-442d-ad40-ec6df1cf620e","Type":"ContainerStarted","Data":"9f67595475c7c4a29355ee95c43ef8922ef2314861a9b843b8cfad5528a997be"}
Oct 29 22:33:36.808107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:36.808071    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq" event={"ID":"d28324a2-e722-4488-81b2-dc8152adaf72","Type":"ContainerStarted","Data":"04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c"}
Oct 29 22:33:36.808232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:36.808125    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq" event={"ID":"d28324a2-e722-4488-81b2-dc8152adaf72","Type":"ContainerStarted","Data":"43bda2fee5f9dd10e6c1cd2acca6569c4075c1d5a4010f4945aa7934dcfaac1f"}
Oct 29 22:33:36.858292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:36.858134    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v" podStartSLOduration=1.858116525 podStartE2EDuration="1.858116525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:36.854098398 +0000 UTC m=+1158.878874529" watchObservedRunningTime="2026-10-29 22:33:36.858116525 +0000 UTC m=+1158.882892657"
Oct 29 22:33:36.926970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:36.926931    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:36.933591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:36.933543    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq" podStartSLOduration=1.9335281370000001 podStartE2EDuration="1.933528137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:36.929926201 +0000 UTC m=+1158.954702352" watchObservedRunningTime="2026-10-29 22:33:36.933528137 +0000 UTC m=+1158.958304330"
Oct 29 22:33:37.032423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.032383    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"]
Oct 29 22:33:37.038309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.038279    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8971ebef-q8tnw"]
Oct 29 22:33:37.098405 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:33:37.096488    4860 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e1dcae_9c4c_442d_ad40_ec6df1cf620e.slice/crio-534e047eb87a69810c7fb09e1a0180b043c5a7abc5596e6e0e3afcd608352499.scope/memory.swap.max": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e1dcae_9c4c_442d_ad40_ec6df1cf620e.slice/crio-534e047eb87a69810c7fb09e1a0180b043c5a7abc5596e6e0e3afcd608352499.scope/memory.swap.max: no such device
Oct 29 22:33:37.098979 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:33:37.098712    4860 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e1dcae_9c4c_442d_ad40_ec6df1cf620e.slice/crio-534e047eb87a69810c7fb09e1a0180b043c5a7abc5596e6e0e3afcd608352499.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:33:37.488415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.482814    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:37.819858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.819767    4860 generic.go:356] "Generic (PLEG): container finished" podID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerID="423001e95419c4a4e48de09fd56b19e65c3941204e573cab6a632b0504f6a3ec" exitCode=0
Oct 29 22:33:37.819858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.819843    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" event={"ID":"8b030dda-fbf0-4d08-800a-d8688aef34c6","Type":"ContainerDied","Data":"423001e95419c4a4e48de09fd56b19e65c3941204e573cab6a632b0504f6a3ec"}
Oct 29 22:33:37.820456 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.820412    4860 scope.go:122] "RemoveContainer" containerID="423001e95419c4a4e48de09fd56b19e65c3941204e573cab6a632b0504f6a3ec"
Oct 29 22:33:37.821857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.821830    4860 generic.go:356] "Generic (PLEG): container finished" podID="41e1dcae-9c4c-442d-ad40-ec6df1cf620e" containerID="534e047eb87a69810c7fb09e1a0180b043c5a7abc5596e6e0e3afcd608352499" exitCode=0
Oct 29 22:33:37.821984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.821892    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v" event={"ID":"41e1dcae-9c4c-442d-ad40-ec6df1cf620e","Type":"ContainerDied","Data":"534e047eb87a69810c7fb09e1a0180b043c5a7abc5596e6e0e3afcd608352499"}
Oct 29 22:33:37.822053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.821974    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:33:37.822123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.822059    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:33:37.822123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.822111    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:33:37.928514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.928482    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:33:37.928670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:37.928531    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq" podUID="d28324a2-e722-4488-81b2-dc8152adaf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 29 22:33:38.116757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:38.116663    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399fff50-fdaf-4acb-9805-ac08d8313079" path="/var/lib/kubelet/pods/399fff50-fdaf-4acb-9805-ac08d8313079/volumes"
Oct 29 22:33:38.834801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:38.834760    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" event={"ID":"8b030dda-fbf0-4d08-800a-d8688aef34c6","Type":"ContainerStarted","Data":"eb6d06f5f35b862c34784bbd3cf109d1a7ea874c1d100b7188ada73597e728a8"}
Oct 29 22:33:38.841188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:38.841169    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:33:38.841281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:38.841245    4860 patch_prober.go:29] interesting pod/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:33:38.841319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:38.841288    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 29 22:33:38.927472 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:38.927428    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:33:38.927686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:38.927495    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq" podUID="d28324a2-e722-4488-81b2-dc8152adaf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 29 22:33:39.190348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.187546    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:39.337781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.337563    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-xks-ca\") pod \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\" (UID: \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\") "
Oct 29 22:33:39.337781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.337764    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-openshift-ca\") pod \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\" (UID: \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\") "
Oct 29 22:33:39.338063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.337823    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-kube-api-access-pt5pc\" (UniqueName: \"kubernetes.io/projected/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-kube-api-access-pt5pc\") pod \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\" (UID: \"41e1dcae-9c4c-442d-ad40-ec6df1cf620e\") "
Oct 29 22:33:39.338302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.338272    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-openshift-ca" pod "41e1dcae-9c4c-442d-ad40-ec6df1cf620e" (UID: "41e1dcae-9c4c-442d-ad40-ec6df1cf620e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:33:39.338689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.338664    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:39.341513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.341485    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-kube-api-access-pt5pc" pod "41e1dcae-9c4c-442d-ad40-ec6df1cf620e" (UID: "41e1dcae-9c4c-442d-ad40-ec6df1cf620e"). InnerVolumeSpecName "kube-api-access-pt5pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:33:39.341680 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.341610    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-xks-ca" pod "41e1dcae-9c4c-442d-ad40-ec6df1cf620e" (UID: "41e1dcae-9c4c-442d-ad40-ec6df1cf620e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:33:39.361353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.361316    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:39.446151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.444001    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:39.446151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.444034    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt5pc\" (UniqueName: \"kubernetes.io/projected/41e1dcae-9c4c-442d-ad40-ec6df1cf620e-kube-api-access-pt5pc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:39.841304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.841225    4860 patch_prober.go:29] interesting pod/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:33:39.841304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.841270    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 29 22:33:39.846165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.846142    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"
Oct 29 22:33:39.846410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.846389    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v" event={"ID":"41e1dcae-9c4c-442d-ad40-ec6df1cf620e","Type":"ContainerDied","Data":"9f67595475c7c4a29355ee95c43ef8922ef2314861a9b843b8cfad5528a997be"}
Oct 29 22:33:39.846517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.846507    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f67595475c7c4a29355ee95c43ef8922ef2314861a9b843b8cfad5528a997be"
Oct 29 22:33:39.927307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.927267    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:33:39.927475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:39.927343    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq" podUID="d28324a2-e722-4488-81b2-dc8152adaf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 29 22:33:40.034681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:40.034650    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"]
Oct 29 22:33:40.040806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:40.040778    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a027f8e-w4gtn"]
Oct 29 22:33:40.127944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:40.127862    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a3308c-f2a3-457d-8771-16f445e50b63" path="/var/lib/kubelet/pods/58a3308c-f2a3-457d-8771-16f445e50b63/volumes"
Oct 29 22:33:40.841010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:40.840979    4860 patch_prober.go:29] interesting pod/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:33:40.841195 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:40.841033    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 29 22:33:40.927275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:40.927241    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:33:40.927712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:40.927357    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq" podUID="d28324a2-e722-4488-81b2-dc8152adaf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 29 22:33:41.089349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:41.086253    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:41.841098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:41.841067    4860 patch_prober.go:29] interesting pod/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:33:41.841271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:41.841115    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 29 22:33:41.927134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:41.927100    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:33:41.927394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:41.927370    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq" podUID="d28324a2-e722-4488-81b2-dc8152adaf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 29 22:33:42.701040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:42.701004    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:42.840852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:42.840816    4860 patch_prober.go:29] interesting pod/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:33:42.841147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:42.841105    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 29 22:33:42.927803 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:42.927767    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:33:42.928218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:42.927830    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq" podUID="d28324a2-e722-4488-81b2-dc8152adaf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 29 22:33:43.235907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.235872    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"]
Oct 29 22:33:43.236631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.236598    4860 cpu_manager.go:414] "containerMap: removing container" podUID="41e1dcae-9c4c-442d-ad40-ec6df1cf620e" containerName="revoke-keys"
Oct 29 22:33:43.236631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.236618    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e1dcae-9c4c-442d-ad40-ec6df1cf620e" containerName="revoke-keys"
Oct 29 22:33:43.236838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.236758    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e1dcae-9c4c-442d-ad40-ec6df1cf620e" containerName="revoke-keys"
Oct 29 22:33:43.242165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.242142    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:43.246674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.246651    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs\" (UID: \"cdfb3e94-26af-4119-aa91-75f342e4a071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:43.246756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.246709    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvhtx\" (UniqueName: \"kubernetes.io/projected/cdfb3e94-26af-4119-aa91-75f342e4a071-kube-api-access-fvhtx\") pod \"maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs\" (UID: \"cdfb3e94-26af-4119-aa91-75f342e4a071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:43.246917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.246896    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs\" (UID: \"cdfb3e94-26af-4119-aa91-75f342e4a071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:43.257383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.257352    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"]
Oct 29 22:33:43.304711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.304681    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:33:43.305403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.304695    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:33:43.305649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.304794    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:33:43.305777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.304864    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:33:43.349621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.349588    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs\" (UID: \"cdfb3e94-26af-4119-aa91-75f342e4a071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:43.349937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.349914    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs\" (UID: \"cdfb3e94-26af-4119-aa91-75f342e4a071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:43.350129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.350111    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvhtx\" (UniqueName: \"kubernetes.io/projected/cdfb3e94-26af-4119-aa91-75f342e4a071-kube-api-access-fvhtx\") pod \"maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs\" (UID: \"cdfb3e94-26af-4119-aa91-75f342e4a071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:43.350564 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.350541    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs\" (UID: \"cdfb3e94-26af-4119-aa91-75f342e4a071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:43.352275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.352251    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs\" (UID: \"cdfb3e94-26af-4119-aa91-75f342e4a071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:43.393346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.391150    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvhtx\" (UniqueName: \"kubernetes.io/projected/cdfb3e94-26af-4119-aa91-75f342e4a071-kube-api-access-fvhtx\") pod \"maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs\" (UID: \"cdfb3e94-26af-4119-aa91-75f342e4a071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:43.559587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.559500    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:43.841040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.840959    4860 patch_prober.go:29] interesting pod/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 29 22:33:43.841040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.841012    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 29 22:33:43.904646 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:33:43.904602    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfb3e94_26af_4119_aa91_75f342e4a071.slice/crio-e7a53d0647ea53bd4e13239f460ce02f9849ccd65d3a59ceb96ec92b0b4638cf WatchSource:0}: Error finding container e7a53d0647ea53bd4e13239f460ce02f9849ccd65d3a59ceb96ec92b0b4638cf: Status 404 returned error can't find the container with id e7a53d0647ea53bd4e13239f460ce02f9849ccd65d3a59ceb96ec92b0b4638cf
Oct 29 22:33:43.913936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.910357    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"]
Oct 29 22:33:43.996649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.996613    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:43.997213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.997191    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:43.998075 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:43.998057    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:33:44.039944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:44.039909    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"]
Oct 29 22:33:44.050279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:44.050247    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a96dc7b9-qx6lv"]
Oct 29 22:33:44.124078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:44.123983    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7314749-483a-4958-9bfd-836488a4f14c" path="/var/lib/kubelet/pods/c7314749-483a-4958-9bfd-836488a4f14c/volumes"
Oct 29 22:33:44.438399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:44.438370    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:44.499046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:44.499017    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:33:44.845083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:44.845060    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:33:44.924723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:44.924693    4860 generic.go:356] "Generic (PLEG): container finished" podID="cdfb3e94-26af-4119-aa91-75f342e4a071" containerID="b02471fcc7ca16b0f69f6bc09deaf70a942bac5b3090f83abffdc62f2f0fb7a1" exitCode=0
Oct 29 22:33:44.924855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:44.924774    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs" event={"ID":"cdfb3e94-26af-4119-aa91-75f342e4a071","Type":"ContainerDied","Data":"b02471fcc7ca16b0f69f6bc09deaf70a942bac5b3090f83abffdc62f2f0fb7a1"}
Oct 29 22:33:44.924855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:44.924832    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs" event={"ID":"cdfb3e94-26af-4119-aa91-75f342e4a071","Type":"ContainerStarted","Data":"e7a53d0647ea53bd4e13239f460ce02f9849ccd65d3a59ceb96ec92b0b4638cf"}
Oct 29 22:33:44.926077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:44.926059    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:33:45.821284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:45.821245    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"]
Oct 29 22:33:45.854074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:45.854038    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"]
Oct 29 22:33:45.994490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:45.994446    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:46.111722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.111687    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:46.156866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.156840    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:46.207727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.207695    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdfb3e94-26af-4119-aa91-75f342e4a071-kube-api-access-fvhtx\" (UniqueName: \"kubernetes.io/projected/cdfb3e94-26af-4119-aa91-75f342e4a071-kube-api-access-fvhtx\") pod \"cdfb3e94-26af-4119-aa91-75f342e4a071\" (UID: \"cdfb3e94-26af-4119-aa91-75f342e4a071\") "
Oct 29 22:33:46.207923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.207828    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-xks-ca\") pod \"cdfb3e94-26af-4119-aa91-75f342e4a071\" (UID: \"cdfb3e94-26af-4119-aa91-75f342e4a071\") "
Oct 29 22:33:46.207989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.207973    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-openshift-ca\") pod \"cdfb3e94-26af-4119-aa91-75f342e4a071\" (UID: \"cdfb3e94-26af-4119-aa91-75f342e4a071\") "
Oct 29 22:33:46.208349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.208306    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-openshift-ca" pod "cdfb3e94-26af-4119-aa91-75f342e4a071" (UID: "cdfb3e94-26af-4119-aa91-75f342e4a071"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:33:46.209202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.209151    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:46.214677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.214649    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdfb3e94-26af-4119-aa91-75f342e4a071-kube-api-access-fvhtx" pod "cdfb3e94-26af-4119-aa91-75f342e4a071" (UID: "cdfb3e94-26af-4119-aa91-75f342e4a071"). InnerVolumeSpecName "kube-api-access-fvhtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:33:46.214857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.214835    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-xks-ca" pod "cdfb3e94-26af-4119-aa91-75f342e4a071" (UID: "cdfb3e94-26af-4119-aa91-75f342e4a071"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:33:46.221541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.221514    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:46.310961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.310923    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfb3e94-26af-4119-aa91-75f342e4a071-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:46.311107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.310982    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvhtx\" (UniqueName: \"kubernetes.io/projected/cdfb3e94-26af-4119-aa91-75f342e4a071-kube-api-access-fvhtx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:46.952362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.952336    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"
Oct 29 22:33:46.952362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.952353    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs" event={"ID":"cdfb3e94-26af-4119-aa91-75f342e4a071","Type":"ContainerDied","Data":"e7a53d0647ea53bd4e13239f460ce02f9849ccd65d3a59ceb96ec92b0b4638cf"}
Oct 29 22:33:46.952807 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:46.952403    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7a53d0647ea53bd4e13239f460ce02f9849ccd65d3a59ceb96ec92b0b4638cf"
Oct 29 22:33:47.676709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:47.676671    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:48.039692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:48.039657    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"]
Oct 29 22:33:48.053411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:48.053379    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ccd153-xkbhf"]
Oct 29 22:33:48.059143 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:33:48.059109    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:33:48.112821 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:48.112794    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa87eb1-b336-4152-a079-ea82d1bcaac4" path="/var/lib/kubelet/pods/1aa87eb1-b336-4152-a079-ea82d1bcaac4/volumes"
Oct 29 22:33:49.057686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:49.057655    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:49.530367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:49.530285    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:49.644788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:49.644751    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"]
Oct 29 22:33:49.662423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:49.662285    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:49.670934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:49.670887    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7dea32d-d8r6v"]
Oct 29 22:33:49.777211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:49.777169    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"]
Oct 29 22:33:49.777487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:49.777457    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy" containerID="cri-o://eb6d06f5f35b862c34784bbd3cf109d1a7ea874c1d100b7188ada73597e728a8" gracePeriod=30
Oct 29 22:33:49.848991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:49.848890    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:50.126293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:50.126211    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e1dcae-9c4c-442d-ad40-ec6df1cf620e" path="/var/lib/kubelet/pods/41e1dcae-9c4c-442d-ad40-ec6df1cf620e/volumes"
Oct 29 22:33:50.475058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:50.475029    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:50.605587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:50.605555    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:50.754073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:50.753995    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:50.821606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:50.821576    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx" podUID="8ef676f8-6c61-44d2-b71f-85a401a4f984" containerName="payload-processing" containerID="cri-o://19a9042e8c22cebf9bcab4d820106cfcd99b73e36cfd6f9b56215f3d754643bf" gracePeriod=25
Oct 29 22:33:50.854530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:50.854498    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr" podUID="04d68764-3b0d-48c9-ad1e-6a8078516fe3" containerName="payload-pre-processing" containerID="cri-o://962c1ec652ba591c4c5eda74466a2d6b3a8d0353c8484ff0b9cd15d931170895" gracePeriod=25
Oct 29 22:33:51.011678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.011590    4860 generic.go:356] "Generic (PLEG): container finished" podID="8ef676f8-6c61-44d2-b71f-85a401a4f984" containerID="19a9042e8c22cebf9bcab4d820106cfcd99b73e36cfd6f9b56215f3d754643bf" exitCode=0
Oct 29 22:33:51.012106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.012075    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx" event={"ID":"8ef676f8-6c61-44d2-b71f-85a401a4f984","Type":"ContainerDied","Data":"19a9042e8c22cebf9bcab4d820106cfcd99b73e36cfd6f9b56215f3d754643bf"}
Oct 29 22:33:51.014012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.013986    4860 generic.go:356] "Generic (PLEG): container finished" podID="04d68764-3b0d-48c9-ad1e-6a8078516fe3" containerID="962c1ec652ba591c4c5eda74466a2d6b3a8d0353c8484ff0b9cd15d931170895" exitCode=0
Oct 29 22:33:51.014130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.014038    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr" event={"ID":"04d68764-3b0d-48c9-ad1e-6a8078516fe3","Type":"ContainerDied","Data":"962c1ec652ba591c4c5eda74466a2d6b3a8d0353c8484ff0b9cd15d931170895"}
Oct 29 22:33:51.135846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.135816    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"
Oct 29 22:33:51.179637 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.179615    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"
Oct 29 22:33:51.185988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.184370    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef676f8-6c61-44d2-b71f-85a401a4f984-kube-api-access-qwwzh\" (UniqueName: \"kubernetes.io/projected/8ef676f8-6c61-44d2-b71f-85a401a4f984-kube-api-access-qwwzh\") pod \"8ef676f8-6c61-44d2-b71f-85a401a4f984\" (UID: \"8ef676f8-6c61-44d2-b71f-85a401a4f984\") "
Oct 29 22:33:51.185988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.184575    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef676f8-6c61-44d2-b71f-85a401a4f984-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef676f8-6c61-44d2-b71f-85a401a4f984-plugins-config-volume\") pod \"8ef676f8-6c61-44d2-b71f-85a401a4f984\" (UID: \"8ef676f8-6c61-44d2-b71f-85a401a4f984\") "
Oct 29 22:33:51.185988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.184966    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef676f8-6c61-44d2-b71f-85a401a4f984-plugins-config-volume" pod "8ef676f8-6c61-44d2-b71f-85a401a4f984" (UID: "8ef676f8-6c61-44d2-b71f-85a401a4f984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:33:51.185988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.185513    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef676f8-6c61-44d2-b71f-85a401a4f984-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:51.203642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.198221    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef676f8-6c61-44d2-b71f-85a401a4f984-kube-api-access-qwwzh" pod "8ef676f8-6c61-44d2-b71f-85a401a4f984" (UID: "8ef676f8-6c61-44d2-b71f-85a401a4f984"). InnerVolumeSpecName "kube-api-access-qwwzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:33:51.235956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.235920    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:51.286190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.286090    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d68764-3b0d-48c9-ad1e-6a8078516fe3-kube-api-access-wpdsz\" (UniqueName: \"kubernetes.io/projected/04d68764-3b0d-48c9-ad1e-6a8078516fe3-kube-api-access-wpdsz\") pod \"04d68764-3b0d-48c9-ad1e-6a8078516fe3\" (UID: \"04d68764-3b0d-48c9-ad1e-6a8078516fe3\") "
Oct 29 22:33:51.286419 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.286400    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d68764-3b0d-48c9-ad1e-6a8078516fe3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04d68764-3b0d-48c9-ad1e-6a8078516fe3-plugins-config-volume\") pod \"04d68764-3b0d-48c9-ad1e-6a8078516fe3\" (UID: \"04d68764-3b0d-48c9-ad1e-6a8078516fe3\") "
Oct 29 22:33:51.287223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.287197    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d68764-3b0d-48c9-ad1e-6a8078516fe3-plugins-config-volume" pod "04d68764-3b0d-48c9-ad1e-6a8078516fe3" (UID: "04d68764-3b0d-48c9-ad1e-6a8078516fe3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:33:51.290444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.288545    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwwzh\" (UniqueName: \"kubernetes.io/projected/8ef676f8-6c61-44d2-b71f-85a401a4f984-kube-api-access-qwwzh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:51.290444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.289845    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d68764-3b0d-48c9-ad1e-6a8078516fe3-kube-api-access-wpdsz" pod "04d68764-3b0d-48c9-ad1e-6a8078516fe3" (UID: "04d68764-3b0d-48c9-ad1e-6a8078516fe3"). InnerVolumeSpecName "kube-api-access-wpdsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:33:51.390657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.390619    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04d68764-3b0d-48c9-ad1e-6a8078516fe3-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:51.390657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.390660    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpdsz\" (UniqueName: \"kubernetes.io/projected/04d68764-3b0d-48c9-ad1e-6a8078516fe3-kube-api-access-wpdsz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:51.480015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:51.479978    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:52.024906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:52.024869    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"
Oct 29 22:33:52.024906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:52.024890    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr" event={"ID":"04d68764-3b0d-48c9-ad1e-6a8078516fe3","Type":"ContainerDied","Data":"06e33cfb6b3a6241c61f4bc6e2d3fc4a8d32625d1ca1f96ae39d76ea5b598699"}
Oct 29 22:33:52.025133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:52.024990    4860 scope.go:122] "RemoveContainer" containerID="962c1ec652ba591c4c5eda74466a2d6b3a8d0353c8484ff0b9cd15d931170895"
Oct 29 22:33:52.026851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:52.026829    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"
Oct 29 22:33:52.026959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:52.026829    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx" event={"ID":"8ef676f8-6c61-44d2-b71f-85a401a4f984","Type":"ContainerDied","Data":"10a86e249d4da77e5e2b9a1e9e8eece7ba22271852057f0decd58b1a5496d4e8"}
Oct 29 22:33:52.040628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:52.040605    4860 scope.go:122] "RemoveContainer" containerID="19a9042e8c22cebf9bcab4d820106cfcd99b73e36cfd6f9b56215f3d754643bf"
Oct 29 22:33:52.073895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:52.073863    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:52.100836 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:52.100750    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"]
Oct 29 22:33:52.118558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:52.116029    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f7dea32d-747d8fcc7d-42zlx"]
Oct 29 22:33:52.130603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:52.130565    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"]
Oct 29 22:33:52.136264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:52.136236    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f7dea32d-79976f6559-8pgqr"]
Oct 29 22:33:52.541586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:52.541558    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:53.254954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:53.254914    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:53.760701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:53.760668    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"]
Oct 29 22:33:53.896806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:53.896774    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:54.018080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:54.017997    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:54.117338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:54.117288    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d68764-3b0d-48c9-ad1e-6a8078516fe3" path="/var/lib/kubelet/pods/04d68764-3b0d-48c9-ad1e-6a8078516fe3/volumes"
Oct 29 22:33:54.117900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:54.117878    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef676f8-6c61-44d2-b71f-85a401a4f984" path="/var/lib/kubelet/pods/8ef676f8-6c61-44d2-b71f-85a401a4f984/volumes"
Oct 29 22:33:55.073142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.073060    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:55.099776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.099597    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" event={"ID":"8b030dda-fbf0-4d08-800a-d8688aef34c6","Type":"ContainerDied","Data":"eb6d06f5f35b862c34784bbd3cf109d1a7ea874c1d100b7188ada73597e728a8"}
Oct 29 22:33:55.099776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.099681    4860 scope.go:122] "RemoveContainer" containerID="423001e95419c4a4e48de09fd56b19e65c3941204e573cab6a632b0504f6a3ec"
Oct 29 22:33:55.100063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.099565    4860 generic.go:356] "Generic (PLEG): container finished" podID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerID="eb6d06f5f35b862c34784bbd3cf109d1a7ea874c1d100b7188ada73597e728a8" exitCode=0
Oct 29 22:33:55.230883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.230478    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:55.245353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.244157    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:33:55.412400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.399361    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-socket\") pod \"8b030dda-fbf0-4d08-800a-d8688aef34c6\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") "
Oct 29 22:33:55.412400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.399425    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-podinfo\") pod \"8b030dda-fbf0-4d08-800a-d8688aef34c6\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") "
Oct 29 22:33:55.412400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.399463    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-kube-api-access-shfzg\" (UniqueName: \"kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-kube-api-access-shfzg\") pod \"8b030dda-fbf0-4d08-800a-d8688aef34c6\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") "
Oct 29 22:33:55.412400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.399497    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-credential-socket\") pod \"8b030dda-fbf0-4d08-800a-d8688aef34c6\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") "
Oct 29 22:33:55.412400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.399538    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-envoy\") pod \"8b030dda-fbf0-4d08-800a-d8688aef34c6\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") "
Oct 29 22:33:55.412400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.399635    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-certs\") pod \"8b030dda-fbf0-4d08-800a-d8688aef34c6\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") "
Oct 29 22:33:55.412400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.399715    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-data\") pod \"8b030dda-fbf0-4d08-800a-d8688aef34c6\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") "
Oct 29 22:33:55.412400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.399780    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-token\" (UniqueName: \"kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-token\") pod \"8b030dda-fbf0-4d08-800a-d8688aef34c6\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") "
Oct 29 22:33:55.412400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.399872    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b030dda-fbf0-4d08-800a-d8688aef34c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b030dda-fbf0-4d08-800a-d8688aef34c6-istiod-ca-cert\") pod \"8b030dda-fbf0-4d08-800a-d8688aef34c6\" (UID: \"8b030dda-fbf0-4d08-800a-d8688aef34c6\") "
Oct 29 22:33:55.466422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.457007    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b030dda-fbf0-4d08-800a-d8688aef34c6-istiod-ca-cert" pod "8b030dda-fbf0-4d08-800a-d8688aef34c6" (UID: "8b030dda-fbf0-4d08-800a-d8688aef34c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:33:55.466422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.457785    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-data" pod "8b030dda-fbf0-4d08-800a-d8688aef34c6" (UID: "8b030dda-fbf0-4d08-800a-d8688aef34c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:33:55.466422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.458218    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-socket" pod "8b030dda-fbf0-4d08-800a-d8688aef34c6" (UID: "8b030dda-fbf0-4d08-800a-d8688aef34c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:33:55.466422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.458278    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-certs" pod "8b030dda-fbf0-4d08-800a-d8688aef34c6" (UID: "8b030dda-fbf0-4d08-800a-d8688aef34c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:33:55.466422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.462274    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-podinfo" pod "8b030dda-fbf0-4d08-800a-d8688aef34c6" (UID: "8b030dda-fbf0-4d08-800a-d8688aef34c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:33:55.518344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.516677    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:55.518344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.516716    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:55.518344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.516732    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:55.518344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.516745    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:55.518344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.516759    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b030dda-fbf0-4d08-800a-d8688aef34c6-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:55.553488 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.553452    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-credential-socket" pod "8b030dda-fbf0-4d08-800a-d8688aef34c6" (UID: "8b030dda-fbf0-4d08-800a-d8688aef34c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:33:55.554589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.554142    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-envoy" pod "8b030dda-fbf0-4d08-800a-d8688aef34c6" (UID: "8b030dda-fbf0-4d08-800a-d8688aef34c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:33:55.556022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.555998    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-token" pod "8b030dda-fbf0-4d08-800a-d8688aef34c6" (UID: "8b030dda-fbf0-4d08-800a-d8688aef34c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:33:55.561626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.561543    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-kube-api-access-shfzg" pod "8b030dda-fbf0-4d08-800a-d8688aef34c6" (UID: "8b030dda-fbf0-4d08-800a-d8688aef34c6"). InnerVolumeSpecName "kube-api-access-shfzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:33:55.630376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.618472    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shfzg\" (UniqueName: \"kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-kube-api-access-shfzg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:55.630376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.618505    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:55.630376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.618521    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:55.630376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:55.618537    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b030dda-fbf0-4d08-800a-d8688aef34c6-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:56.128420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.127025    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:56.128420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.127224    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"
Oct 29 22:33:56.135592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.135552    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr" event={"ID":"8b030dda-fbf0-4d08-800a-d8688aef34c6","Type":"ContainerDied","Data":"d7a7cadefc528d2314a4ebe9b79b95698bfc14d33524ebd154afe0320ac19baa"}
Oct 29 22:33:56.136602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.136579    4860 scope.go:122] "RemoveContainer" containerID="eb6d06f5f35b862c34784bbd3cf109d1a7ea874c1d100b7188ada73597e728a8"
Oct 29 22:33:56.207736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.207705    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"]
Oct 29 22:33:56.220885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.220848    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f7dea32d-openshift-default-5cf5f998fc-dt4cr"]
Oct 29 22:33:56.263214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.263185    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"]
Oct 29 22:33:56.263823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.263810    4860 cpu_manager.go:414] "containerMap: removing container" podUID="cdfb3e94-26af-4119-aa91-75f342e4a071" containerName="revoke-keys"
Oct 29 22:33:56.263876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.263825    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdfb3e94-26af-4119-aa91-75f342e4a071" containerName="revoke-keys"
Oct 29 22:33:56.263876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.263840    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy"
Oct 29 22:33:56.263876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.263845    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy"
Oct 29 22:33:56.263876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.263852    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy"
Oct 29 22:33:56.263876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.263857    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy"
Oct 29 22:33:56.263876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.263876    4860 cpu_manager.go:414] "containerMap: removing container" podUID="04d68764-3b0d-48c9-ad1e-6a8078516fe3" containerName="payload-pre-processing"
Oct 29 22:33:56.264055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.263881    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d68764-3b0d-48c9-ad1e-6a8078516fe3" containerName="payload-pre-processing"
Oct 29 22:33:56.264055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.263900    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8ef676f8-6c61-44d2-b71f-85a401a4f984" containerName="payload-processing"
Oct 29 22:33:56.264055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.263904    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef676f8-6c61-44d2-b71f-85a401a4f984" containerName="payload-processing"
Oct 29 22:33:56.264055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.264021    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d68764-3b0d-48c9-ad1e-6a8078516fe3" containerName="payload-pre-processing"
Oct 29 22:33:56.264055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.264032    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef676f8-6c61-44d2-b71f-85a401a4f984" containerName="payload-processing"
Oct 29 22:33:56.264055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.264043    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdfb3e94-26af-4119-aa91-75f342e4a071" containerName="revoke-keys"
Oct 29 22:33:56.264055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.264051    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy"
Oct 29 22:33:56.264055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.264057    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" containerName="istio-proxy"
Oct 29 22:33:56.269285 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.269265    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"
Oct 29 22:33:56.291480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.291457    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:56.291626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.291613    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e385da9b-dockercfg-bwmz6\""
Oct 29 22:33:56.296562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.296541    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e385da9b\""
Oct 29 22:33:56.299362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.299344    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"]
Oct 29 22:33:56.372569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.372539    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"]
Oct 29 22:33:56.403442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.380580    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"
Oct 29 22:33:56.446132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.444901    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d65t\" (UniqueName: \"kubernetes.io/projected/e77feed5-cf09-42d0-9ab8-b9ff37753805-kube-api-access-2d65t\") pod \"payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw\" (UID: \"e77feed5-cf09-42d0-9ab8-b9ff37753805\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"
Oct 29 22:33:56.446132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.445119    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77feed5-cf09-42d0-9ab8-b9ff37753805-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw\" (UID: \"e77feed5-cf09-42d0-9ab8-b9ff37753805\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"
Oct 29 22:33:56.459392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.455536    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"]
Oct 29 22:33:56.546396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.546360    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d65t\" (UniqueName: \"kubernetes.io/projected/e77feed5-cf09-42d0-9ab8-b9ff37753805-kube-api-access-2d65t\") pod \"payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw\" (UID: \"e77feed5-cf09-42d0-9ab8-b9ff37753805\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"
Oct 29 22:33:56.546660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.546416    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/177e7be8-f825-44b8-a455-335a0039198a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr\" (UID: \"177e7be8-f825-44b8-a455-335a0039198a\") " pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"
Oct 29 22:33:56.546660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.546635    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77feed5-cf09-42d0-9ab8-b9ff37753805-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw\" (UID: \"e77feed5-cf09-42d0-9ab8-b9ff37753805\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"
Oct 29 22:33:56.546803 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.546669    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjrvz\" (UniqueName: \"kubernetes.io/projected/177e7be8-f825-44b8-a455-335a0039198a-kube-api-access-cjrvz\") pod \"payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr\" (UID: \"177e7be8-f825-44b8-a455-335a0039198a\") " pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"
Oct 29 22:33:56.547475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.547449    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77feed5-cf09-42d0-9ab8-b9ff37753805-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw\" (UID: \"e77feed5-cf09-42d0-9ab8-b9ff37753805\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"
Oct 29 22:33:56.571562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.571528    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d65t\" (UniqueName: \"kubernetes.io/projected/e77feed5-cf09-42d0-9ab8-b9ff37753805-kube-api-access-2d65t\") pod \"payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw\" (UID: \"e77feed5-cf09-42d0-9ab8-b9ff37753805\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"
Oct 29 22:33:56.584429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.584176    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"
Oct 29 22:33:56.648033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.647999    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrvz\" (UniqueName: \"kubernetes.io/projected/177e7be8-f825-44b8-a455-335a0039198a-kube-api-access-cjrvz\") pod \"payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr\" (UID: \"177e7be8-f825-44b8-a455-335a0039198a\") " pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"
Oct 29 22:33:56.648158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.648140    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/177e7be8-f825-44b8-a455-335a0039198a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr\" (UID: \"177e7be8-f825-44b8-a455-335a0039198a\") " pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"
Oct 29 22:33:56.648957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.648923    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/177e7be8-f825-44b8-a455-335a0039198a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr\" (UID: \"177e7be8-f825-44b8-a455-335a0039198a\") " pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"
Oct 29 22:33:56.681242 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.681207    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjrvz\" (UniqueName: \"kubernetes.io/projected/177e7be8-f825-44b8-a455-335a0039198a-kube-api-access-cjrvz\") pod \"payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr\" (UID: \"177e7be8-f825-44b8-a455-335a0039198a\") " pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"
Oct 29 22:33:56.775042 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.775004    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"
Oct 29 22:33:56.898195 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:33:56.898155    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77feed5_cf09_42d0_9ab8_b9ff37753805.slice/crio-9bf453198bd0adcda4c4906535139163ffb45ff90a9f62b150f4f1156b0318af WatchSource:0}: Error finding container 9bf453198bd0adcda4c4906535139163ffb45ff90a9f62b150f4f1156b0318af: Status 404 returned error can't find the container with id 9bf453198bd0adcda4c4906535139163ffb45ff90a9f62b150f4f1156b0318af
Oct 29 22:33:56.904559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:56.904511    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"]
Oct 29 22:33:57.034112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.034086    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"]
Oct 29 22:33:57.041095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.041059    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"]
Oct 29 22:33:57.046127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.046102    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c7a8ca-zgnn7"]
Oct 29 22:33:57.051213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.051181    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d8ffb13-j7n8x"]
Oct 29 22:33:57.077051 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:33:57.077022    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177e7be8_f825_44b8_a455_335a0039198a.slice/crio-010ae2bd33880c5698271e98536f9bb9170382fa9d2edd698498f06d078567a5 WatchSource:0}: Error finding container 010ae2bd33880c5698271e98536f9bb9170382fa9d2edd698498f06d078567a5: Status 404 returned error can't find the container with id 010ae2bd33880c5698271e98536f9bb9170382fa9d2edd698498f06d078567a5
Oct 29 22:33:57.080489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.080473    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"]
Oct 29 22:33:57.138880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.138845    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr" event={"ID":"177e7be8-f825-44b8-a455-335a0039198a","Type":"ContainerStarted","Data":"010ae2bd33880c5698271e98536f9bb9170382fa9d2edd698498f06d078567a5"}
Oct 29 22:33:57.142262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.142229    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw" event={"ID":"e77feed5-cf09-42d0-9ab8-b9ff37753805","Type":"ContainerStarted","Data":"54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39"}
Oct 29 22:33:57.142419 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.142276    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw" event={"ID":"e77feed5-cf09-42d0-9ab8-b9ff37753805","Type":"ContainerStarted","Data":"9bf453198bd0adcda4c4906535139163ffb45ff90a9f62b150f4f1156b0318af"}
Oct 29 22:33:57.142419 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.142366    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"
Oct 29 22:33:57.185407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.185361    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw" podStartSLOduration=1.185315435 podStartE2EDuration="1.185315435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:57.184727136 +0000 UTC m=+1179.209503280" watchObservedRunningTime="2026-10-29 22:33:57.185315435 +0000 UTC m=+1179.210091608"
Oct 29 22:33:57.210533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.210501    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:57.818217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.818175    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"]
Oct 29 22:33:57.822973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.822951    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:33:57.840825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.839175    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:33:57.840825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.839214    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:33:57.841746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.841722    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"]
Oct 29 22:33:57.848537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.848519    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:33:57.848646 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.848539    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:33:57.864206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.864183    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc893314-qkwqs\" (UID: \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:33:57.864363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.864342    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs4ms\" (UniqueName: \"kubernetes.io/projected/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-kube-api-access-zs4ms\") pod \"maas-api-revoke-sub-bc893314-qkwqs\" (UID: \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:33:57.864442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.864387    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc893314-qkwqs\" (UID: \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:33:57.965368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.965341    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc893314-qkwqs\" (UID: \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:33:57.965496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.965471    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs4ms\" (UniqueName: \"kubernetes.io/projected/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-kube-api-access-zs4ms\") pod \"maas-api-revoke-sub-bc893314-qkwqs\" (UID: \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:33:57.965539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.965524    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc893314-qkwqs\" (UID: \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:33:57.966021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.966001    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc893314-qkwqs\" (UID: \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:33:57.968054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.968024    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc893314-qkwqs\" (UID: \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:33:57.987928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:57.987903    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs4ms\" (UniqueName: \"kubernetes.io/projected/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-kube-api-access-zs4ms\") pod \"maas-api-revoke-sub-bc893314-qkwqs\" (UID: \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:33:58.116158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:58.116085    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330b7223-95fc-412f-8825-547da318f55f" path="/var/lib/kubelet/pods/330b7223-95fc-412f-8825-547da318f55f/volumes"
Oct 29 22:33:58.116574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:58.116559    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a60753-01c7-4486-b99f-6d3f2178f07e" path="/var/lib/kubelet/pods/78a60753-01c7-4486-b99f-6d3f2178f07e/volumes"
Oct 29 22:33:58.117013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:58.117000    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b030dda-fbf0-4d08-800a-d8688aef34c6" path="/var/lib/kubelet/pods/8b030dda-fbf0-4d08-800a-d8688aef34c6/volumes"
Oct 29 22:33:58.137528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:58.137498    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:33:58.160530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:58.160496    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr" event={"ID":"177e7be8-f825-44b8-a455-335a0039198a","Type":"ContainerStarted","Data":"ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be"}
Oct 29 22:33:58.160942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:58.160585    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"
Oct 29 22:33:58.207779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:58.207732    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr" podStartSLOduration=2.20771735 podStartE2EDuration="2.20771735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:58.206176554 +0000 UTC m=+1180.230952706" watchObservedRunningTime="2026-10-29 22:33:58.20771735 +0000 UTC m=+1180.232493481"
Oct 29 22:33:58.447675 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:33:58.447645    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcb5a35_7e4a_43f7_98b5_05bf3b35acdc.slice/crio-da84e2ce3c8bc9579101029f1414bbd68f5fd2634a0bd043a562011e49268996 WatchSource:0}: Error finding container da84e2ce3c8bc9579101029f1414bbd68f5fd2634a0bd043a562011e49268996: Status 404 returned error can't find the container with id da84e2ce3c8bc9579101029f1414bbd68f5fd2634a0bd043a562011e49268996
Oct 29 22:33:58.452877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:58.452849    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"]
Oct 29 22:33:58.761223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:58.761156    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch" podUID="86650176-e767-4503-8a16-43c9761da511" containerName="payload-pre-processing" containerID="cri-o://b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5" gracePeriod=25
Oct 29 22:33:58.939853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:58.939821    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:33:59.078365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.078317    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"
Oct 29 22:33:59.185298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.185256    4860 generic.go:356] "Generic (PLEG): container finished" podID="86650176-e767-4503-8a16-43c9761da511" containerID="b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5" exitCode=0
Oct 29 22:33:59.185879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.185319    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"
Oct 29 22:33:59.185879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.185588    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch" event={"ID":"86650176-e767-4503-8a16-43c9761da511","Type":"ContainerDied","Data":"b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5"}
Oct 29 22:33:59.185879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.185670    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch" event={"ID":"86650176-e767-4503-8a16-43c9761da511","Type":"ContainerDied","Data":"fc4aa22737e0f516f64ba07a999e6e830b6df4aaee27095a2a1c9f4b0fcdfc3d"}
Oct 29 22:33:59.185879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.185701    4860 scope.go:122] "RemoveContainer" containerID="b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5"
Oct 29 22:33:59.189619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.189591    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs" event={"ID":"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc","Type":"ContainerStarted","Data":"dd1a12f3ba1bdff7bc3bd78cd1e1a540d09052b1402979d304c8ebd7a5ad85de"}
Oct 29 22:33:59.189733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.189641    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs" event={"ID":"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc","Type":"ContainerStarted","Data":"da84e2ce3c8bc9579101029f1414bbd68f5fd2634a0bd043a562011e49268996"}
Oct 29 22:33:59.198392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.198377    4860 scope.go:122] "RemoveContainer" containerID="b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5"
Oct 29 22:33:59.198682 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:33:59.198663    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5\": container with ID starting with b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5 not found: ID does not exist" containerID="b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5"
Oct 29 22:33:59.198778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.198686    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5"} err="failed to get container status \"b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5\": rpc error: code = NotFound desc = could not find container \"b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5\": container with ID starting with b24217d5b59a202127bd0f7f70751bea0466ff73456dd85ce47a034ed579a5c5 not found: ID does not exist"
Oct 29 22:33:59.208861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.208838    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86650176-e767-4503-8a16-43c9761da511-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86650176-e767-4503-8a16-43c9761da511-plugins-config-volume\") pod \"86650176-e767-4503-8a16-43c9761da511\" (UID: \"86650176-e767-4503-8a16-43c9761da511\") "
Oct 29 22:33:59.209065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.209045    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86650176-e767-4503-8a16-43c9761da511-kube-api-access-fnclh\" (UniqueName: \"kubernetes.io/projected/86650176-e767-4503-8a16-43c9761da511-kube-api-access-fnclh\") pod \"86650176-e767-4503-8a16-43c9761da511\" (UID: \"86650176-e767-4503-8a16-43c9761da511\") "
Oct 29 22:33:59.209156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.209102    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86650176-e767-4503-8a16-43c9761da511-plugins-config-volume" pod "86650176-e767-4503-8a16-43c9761da511" (UID: "86650176-e767-4503-8a16-43c9761da511"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:33:59.209447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.209431    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86650176-e767-4503-8a16-43c9761da511-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:59.211060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.211036    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86650176-e767-4503-8a16-43c9761da511-kube-api-access-fnclh" pod "86650176-e767-4503-8a16-43c9761da511" (UID: "86650176-e767-4503-8a16-43c9761da511"). InnerVolumeSpecName "kube-api-access-fnclh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:33:59.243889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.243845    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs" podStartSLOduration=2.243830568 podStartE2EDuration="2.243830568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:59.242788649 +0000 UTC m=+1181.267564779" watchObservedRunningTime="2026-10-29 22:33:59.243830568 +0000 UTC m=+1181.268606700"
Oct 29 22:33:59.310937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.310873    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnclh\" (UniqueName: \"kubernetes.io/projected/86650176-e767-4503-8a16-43c9761da511-kube-api-access-fnclh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:33:59.536875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.536850    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"]
Oct 29 22:33:59.550104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:33:59.550071    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc369ed0-558776d6ff-bhzch"]
Oct 29 22:34:00.113608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:00.113574    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86650176-e767-4503-8a16-43c9761da511" path="/var/lib/kubelet/pods/86650176-e767-4503-8a16-43c9761da511/volumes"
Oct 29 22:34:00.198337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:00.198298    4860 generic.go:356] "Generic (PLEG): container finished" podID="fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc" containerID="dd1a12f3ba1bdff7bc3bd78cd1e1a540d09052b1402979d304c8ebd7a5ad85de" exitCode=0
Oct 29 22:34:00.198739 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:00.198388    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs" event={"ID":"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc","Type":"ContainerDied","Data":"dd1a12f3ba1bdff7bc3bd78cd1e1a540d09052b1402979d304c8ebd7a5ad85de"}
Oct 29 22:34:00.534962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:00.534918    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:01.031872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:01.031836    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"]
Oct 29 22:34:01.045926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:01.045898    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157cfc60-csphq"]
Oct 29 22:34:01.372898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:01.372874    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:34:01.439508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:01.439483    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-xks-ca\") pod \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\" (UID: \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\") "
Oct 29 22:34:01.439647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:01.439521    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-openshift-ca\") pod \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\" (UID: \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\") "
Oct 29 22:34:01.439791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:01.439771    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-kube-api-access-zs4ms\" (UniqueName: \"kubernetes.io/projected/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-kube-api-access-zs4ms\") pod \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\" (UID: \"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc\") "
Oct 29 22:34:01.439964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:01.439931    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-openshift-ca" pod "fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc" (UID: "fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:34:01.440481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:01.440456    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:01.441744 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:01.441720    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-xks-ca" pod "fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc" (UID: "fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:34:01.441945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:01.441920    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-kube-api-access-zs4ms" pod "fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc" (UID: "fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc"). InnerVolumeSpecName "kube-api-access-zs4ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:34:01.541801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:01.541775    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs4ms\" (UniqueName: \"kubernetes.io/projected/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-kube-api-access-zs4ms\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:01.541801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:01.541800    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:02.066509 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:02.066480    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:02.115065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:02.115028    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc7d10b-c324-4222-b766-d88a2548f84c" path="/var/lib/kubelet/pods/dfc7d10b-c324-4222-b766-d88a2548f84c/volumes"
Oct 29 22:34:02.219968 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:02.219940    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"
Oct 29 22:34:02.220134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:02.219942    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs" event={"ID":"fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc","Type":"ContainerDied","Data":"da84e2ce3c8bc9579101029f1414bbd68f5fd2634a0bd043a562011e49268996"}
Oct 29 22:34:02.220134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:02.220059    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da84e2ce3c8bc9579101029f1414bbd68f5fd2634a0bd043a562011e49268996"
Oct 29 22:34:03.749339 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:03.749303    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:05.036890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.036858    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"]
Oct 29 22:34:05.050806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.050776    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"]
Oct 29 22:34:05.056138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.056116    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0eb50da-r7fp6"]
Oct 29 22:34:05.061365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.061345    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae35e6a-728l8"]
Oct 29 22:34:05.270590 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.270558    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:05.620905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.620875    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"]
Oct 29 22:34:05.621896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.621873    4860 cpu_manager.go:414] "containerMap: removing container" podUID="86650176-e767-4503-8a16-43c9761da511" containerName="payload-pre-processing"
Oct 29 22:34:05.621896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.621895    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86650176-e767-4503-8a16-43c9761da511" containerName="payload-pre-processing"
Oct 29 22:34:05.622112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.621907    4860 cpu_manager.go:414] "containerMap: removing container" podUID="fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc" containerName="revoke-keys"
Oct 29 22:34:05.622112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.621912    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc" containerName="revoke-keys"
Oct 29 22:34:05.622112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.622030    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="86650176-e767-4503-8a16-43c9761da511" containerName="payload-pre-processing"
Oct 29 22:34:05.622112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.622040    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc" containerName="revoke-keys"
Oct 29 22:34:05.626747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.626726    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.647589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.647565    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"]
Oct 29 22:34:05.698683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.698651    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.698816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.698693    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.698816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.698718    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72edf772-cda9-4f90-8725-4913072cc67c-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.698960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.698856    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.698960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.698930    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.699088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.699071    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72edf772-cda9-4f90-8725-4913072cc67c-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.699147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.699114    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n98p5\" (UniqueName: \"kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-kube-api-access-n98p5\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.699222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.699206    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.699263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.699244    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.800301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.800257    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.800452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.800314    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.800452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.800374    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72edf772-cda9-4f90-8725-4913072cc67c-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.800452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.800407    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.800644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.800592    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.800709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.800664    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-data\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.800754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.800726    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72edf772-cda9-4f90-8725-4913072cc67c-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.800809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.800762    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98p5\" (UniqueName: \"kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-kube-api-access-n98p5\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.800869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.800852    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.800933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.800904    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.800986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.800934    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-certs\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.801084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.801061    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72edf772-cda9-4f90-8725-4913072cc67c-istiod-ca-cert\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.801266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.801245    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.801310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.801275    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-credential-socket\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.802823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.802806    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-envoy\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.803090 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.803074    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72edf772-cda9-4f90-8725-4913072cc67c-istio-podinfo\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.820957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.820938    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-istio-token\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.821128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.821111    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98p5\" (UniqueName: \"kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-kube-api-access-n98p5\") pod \"e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") " pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:05.946423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:05.946398    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:06.125338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:06.125294    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0debd5d9-d5f0-4c81-b35f-696613811010" path="/var/lib/kubelet/pods/0debd5d9-d5f0-4c81-b35f-696613811010/volumes"
Oct 29 22:34:06.125797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:06.125782    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="729aaaa7-6766-4d22-84c0-88bc54b2b74a" path="/var/lib/kubelet/pods/729aaaa7-6766-4d22-84c0-88bc54b2b74a/volumes"
Oct 29 22:34:06.266373 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:34:06.266345    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72edf772_cda9_4f90_8725_4913072cc67c.slice/crio-36b7de2eb08aa7e0798e6db7e0d43b71854f5b6cd9e8b7ba81ee9614cf8bb6bd WatchSource:0}: Error finding container 36b7de2eb08aa7e0798e6db7e0d43b71854f5b6cd9e8b7ba81ee9614cf8bb6bd: Status 404 returned error can't find the container with id 36b7de2eb08aa7e0798e6db7e0d43b71854f5b6cd9e8b7ba81ee9614cf8bb6bd
Oct 29 22:34:06.268795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:06.268761    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:34:06.268868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:06.268830    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:34:06.268906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:06.268873    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:34:06.271298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:06.271275    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"]
Oct 29 22:34:06.694290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:06.694254    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:07.179223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:07.179127    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:07.324349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:07.323964    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:07.329359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:07.326538    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5" event={"ID":"72edf772-cda9-4f90-8725-4913072cc67c","Type":"ContainerStarted","Data":"739765837b1a023c885f79404d14deab999fa12895edcba4008591377b55eeb5"}
Oct 29 22:34:07.329359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:07.326601    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5" event={"ID":"72edf772-cda9-4f90-8725-4913072cc67c","Type":"ContainerStarted","Data":"36b7de2eb08aa7e0798e6db7e0d43b71854f5b6cd9e8b7ba81ee9614cf8bb6bd"}
Oct 29 22:34:07.346932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:07.346899    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"]
Oct 29 22:34:07.356758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:07.356659    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc369ed0-6hfvs"]
Oct 29 22:34:07.399491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:07.399404    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5" podStartSLOduration=2.399382471 podStartE2EDuration="2.399382471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:07.37581751 +0000 UTC m=+1189.400593642" watchObservedRunningTime="2026-10-29 22:34:07.399382471 +0000 UTC m=+1189.424158603"
Oct 29 22:34:07.947194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:07.947158    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:07.949217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:07.949187    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 29 22:34:07.949395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:07.949232    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5" podUID="72edf772-cda9-4f90-8725-4913072cc67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 29 22:34:08.124220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.124183    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdfb3e94-26af-4119-aa91-75f342e4a071" path="/var/lib/kubelet/pods/cdfb3e94-26af-4119-aa91-75f342e4a071/volumes"
Oct 29 22:34:08.164255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.164217    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"
Oct 29 22:34:08.220033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.219957    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"]
Oct 29 22:34:08.226337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.226299    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:08.243603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.243581    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:34:08.243766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.243622    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:34:08.244028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.243715    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:34:08.244151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.244124    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:34:08.247905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.247882    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"]
Oct 29 22:34:08.343772 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.343734    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-418d5d3e-2clvz\" (UID: \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:08.343922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.343795    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-418d5d3e-2clvz\" (UID: \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:08.344011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.343997    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wlkj\" (UniqueName: \"kubernetes.io/projected/bc7454f3-4494-41b6-a10d-1d85ca7646f9-kube-api-access-6wlkj\") pod \"maas-api-revoke-sub-418d5d3e-2clvz\" (UID: \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:08.445388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.445354    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-418d5d3e-2clvz\" (UID: \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:08.445575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.445433    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-418d5d3e-2clvz\" (UID: \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:08.445735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.445705    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wlkj\" (UniqueName: \"kubernetes.io/projected/bc7454f3-4494-41b6-a10d-1d85ca7646f9-kube-api-access-6wlkj\") pod \"maas-api-revoke-sub-418d5d3e-2clvz\" (UID: \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:08.446213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.446187    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-418d5d3e-2clvz\" (UID: \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:08.447962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.447939    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-418d5d3e-2clvz\" (UID: \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:08.485379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.467525    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wlkj\" (UniqueName: \"kubernetes.io/projected/bc7454f3-4494-41b6-a10d-1d85ca7646f9-kube-api-access-6wlkj\") pod \"maas-api-revoke-sub-418d5d3e-2clvz\" (UID: \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:08.550622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.550599    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:08.904236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.904205    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"]
Oct 29 22:34:08.907254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.907218    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:08.947560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.947533    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 29 22:34:08.947664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:08.947608    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5" podUID="72edf772-cda9-4f90-8725-4913072cc67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 29 22:34:09.199574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:09.199542    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"
Oct 29 22:34:09.352177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:09.352138    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz" event={"ID":"bc7454f3-4494-41b6-a10d-1d85ca7646f9","Type":"ContainerStarted","Data":"7a8324ba984eeb3e79c31c6734f246fdfacfb12c11c9283ff88eeb1966eb869b"}
Oct 29 22:34:09.352758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:09.352730    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz" event={"ID":"bc7454f3-4494-41b6-a10d-1d85ca7646f9","Type":"ContainerStarted","Data":"b9a56b456e9347ba35e5aa822cd9e51c96a8df5732bfb723db750c8ad737679b"}
Oct 29 22:34:09.400426 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:09.400387    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:09.401006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:09.400964    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz" podStartSLOduration=1.400950273 podStartE2EDuration="1.400950273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:09.398741057 +0000 UTC m=+1191.423517202" watchObservedRunningTime="2026-10-29 22:34:09.400950273 +0000 UTC m=+1191.425726404"
Oct 29 22:34:09.521503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:09.521419    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:09.653643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:09.653602    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:09.947659 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:09.947624    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 29 22:34:09.947814 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:09.947670    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5" podUID="72edf772-cda9-4f90-8725-4913072cc67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 29 22:34:10.004854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.004816    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"]
Oct 29 22:34:10.011053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.011026    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.021014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.020987    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"]
Oct 29 22:34:10.028641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.028621    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6c6e65e5-openshift-default-dockercfg-647rw\""
Oct 29 22:34:10.041969 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.041949    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:10.159694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.159664    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:10.198499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.198409    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-credential-socket\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.199890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.198883    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-token\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.199890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.199219    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpm6v\" (UniqueName: \"kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-kube-api-access-lpm6v\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.199890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.199380    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-socket\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.199890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.199426    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-envoy\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.199890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.199471    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-podinfo\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.199890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.199503    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istiod-ca-cert\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.199890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.199527    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-data\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.199890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.199627    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-certs\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.303458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.303398    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpm6v\" (UniqueName: \"kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-kube-api-access-lpm6v\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.305532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.305501    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-socket\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.305764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.305744    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-envoy\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.307346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.305897    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-podinfo\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.307549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.307518    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istiod-ca-cert\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.307644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.307563    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-data\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.307735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.307719    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-certs\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.307828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.307812    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-credential-socket\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.307872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.307858    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-token\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.308895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.308872    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-socket\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.310655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.310631    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-data\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.311252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.310901    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-certs\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.311252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.311191    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-credential-socket\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.312287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.312264    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istiod-ca-cert\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.313064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.313038    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-envoy\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.313210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.313193    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-podinfo\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.331343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.329891    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpm6v\" (UniqueName: \"kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-kube-api-access-lpm6v\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.333378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.331936    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-token\") pod \"e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") " pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.364691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.364661    4860 generic.go:356] "Generic (PLEG): container finished" podID="bc7454f3-4494-41b6-a10d-1d85ca7646f9" containerID="7a8324ba984eeb3e79c31c6734f246fdfacfb12c11c9283ff88eeb1966eb869b" exitCode=0
Oct 29 22:34:10.365106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.364695    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz" event={"ID":"bc7454f3-4494-41b6-a10d-1d85ca7646f9","Type":"ContainerDied","Data":"7a8324ba984eeb3e79c31c6734f246fdfacfb12c11c9283ff88eeb1966eb869b"}
Oct 29 22:34:10.632106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.632021    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:10.947624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.947587    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 29 22:34:10.947774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:10.947648    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5" podUID="72edf772-cda9-4f90-8725-4913072cc67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 29 22:34:11.049788 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:34:11.049747    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb808e4d2_a76c_4798_ae4d_74bbf0020fd5.slice/crio-bc0dea5e0dd88afef894d5de67368707eb767ad53e5a8e5def647b4a522829f3 WatchSource:0}: Error finding container bc0dea5e0dd88afef894d5de67368707eb767ad53e5a8e5def647b4a522829f3: Status 404 returned error can't find the container with id bc0dea5e0dd88afef894d5de67368707eb767ad53e5a8e5def647b4a522829f3
Oct 29 22:34:11.053981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.053938    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:34:11.054091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.054026    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:34:11.054091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.054069    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:34:11.055666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.055642    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"]
Oct 29 22:34:11.196981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.195607    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:11.326313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.325811    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:11.400948 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.400907    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz" event={"ID":"b808e4d2-a76c-4798-ae4d-74bbf0020fd5","Type":"ContainerStarted","Data":"984776f21e7d5e196baf8de5a629c317f813493b574d30d0808d13e1debf2147"}
Oct 29 22:34:11.401491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.400966    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz" event={"ID":"b808e4d2-a76c-4798-ae4d-74bbf0020fd5","Type":"ContainerStarted","Data":"bc0dea5e0dd88afef894d5de67368707eb767ad53e5a8e5def647b4a522829f3"}
Oct 29 22:34:11.446103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.446052    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz" podStartSLOduration=2.446036881 podStartE2EDuration="2.446036881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:11.445108742 +0000 UTC m=+1193.469884915" watchObservedRunningTime="2026-10-29 22:34:11.446036881 +0000 UTC m=+1193.470813012"
Oct 29 22:34:11.637121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.635416    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:11.659109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.654607    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:11.736839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.736751    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-openshift-ca\") pod \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\" (UID: \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\") "
Oct 29 22:34:11.737014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.736878    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-xks-ca\") pod \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\" (UID: \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\") "
Oct 29 22:34:11.737014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.736942    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7454f3-4494-41b6-a10d-1d85ca7646f9-kube-api-access-6wlkj\" (UniqueName: \"kubernetes.io/projected/bc7454f3-4494-41b6-a10d-1d85ca7646f9-kube-api-access-6wlkj\") pod \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\" (UID: \"bc7454f3-4494-41b6-a10d-1d85ca7646f9\") "
Oct 29 22:34:11.737773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.737739    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-openshift-ca" pod "bc7454f3-4494-41b6-a10d-1d85ca7646f9" (UID: "bc7454f3-4494-41b6-a10d-1d85ca7646f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:34:11.739889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.739856    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7454f3-4494-41b6-a10d-1d85ca7646f9-kube-api-access-6wlkj" pod "bc7454f3-4494-41b6-a10d-1d85ca7646f9" (UID: "bc7454f3-4494-41b6-a10d-1d85ca7646f9"). InnerVolumeSpecName "kube-api-access-6wlkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:34:11.740101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.740072    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-xks-ca" pod "bc7454f3-4494-41b6-a10d-1d85ca7646f9" (UID: "bc7454f3-4494-41b6-a10d-1d85ca7646f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:34:11.838257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.838226    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:11.838257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.838256    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7454f3-4494-41b6-a10d-1d85ca7646f9-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:11.838257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.838266    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wlkj\" (UniqueName: \"kubernetes.io/projected/bc7454f3-4494-41b6-a10d-1d85ca7646f9-kube-api-access-6wlkj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:11.947453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.947421    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 29 22:34:11.947655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:11.947478    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5" podUID="72edf772-cda9-4f90-8725-4913072cc67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 29 22:34:12.411571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:12.411493    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"
Oct 29 22:34:12.411571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:12.411515    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz" event={"ID":"bc7454f3-4494-41b6-a10d-1d85ca7646f9","Type":"ContainerDied","Data":"b9a56b456e9347ba35e5aa822cd9e51c96a8df5732bfb723db750c8ad737679b"}
Oct 29 22:34:12.411571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:12.411569    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9a56b456e9347ba35e5aa822cd9e51c96a8df5732bfb723db750c8ad737679b"
Oct 29 22:34:12.639244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:12.639215    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:12.947276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:12.947241    4860 patch_prober.go:29] interesting pod/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 29 22:34:12.947494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:12.947362    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5" podUID="72edf772-cda9-4f90-8725-4913072cc67c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 29 22:34:12.970858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:12.970825    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:13.115801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.115766    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:13.250246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.250161    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:13.422499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.422458    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:13.423316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.423296    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:34:13.935822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.935787    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"]
Oct 29 22:34:13.936887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.936861    4860 cpu_manager.go:414] "containerMap: removing container" podUID="bc7454f3-4494-41b6-a10d-1d85ca7646f9" containerName="revoke-keys"
Oct 29 22:34:13.936887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.936887    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7454f3-4494-41b6-a10d-1d85ca7646f9" containerName="revoke-keys"
Oct 29 22:34:13.937078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.937065    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7454f3-4494-41b6-a10d-1d85ca7646f9" containerName="revoke-keys"
Oct 29 22:34:13.941307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.941234    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:13.967302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.967024    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tmp-dir\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:13.967302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.967144    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-dshm\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:13.967302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.967177    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-model-cache\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:13.967302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.967200    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tls-certs\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:13.967302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.967264    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-home\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:13.967623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.967314    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhjp\" (UniqueName: \"kubernetes.io/projected/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-kube-api-access-7bhjp\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:13.967623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:13.967461    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"]
Oct 29 22:34:14.010315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.010285    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fc684b\"/\"distinct-w1-fc684b-kserve-self-signed-certs\""
Oct 29 22:34:14.018956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.018924    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:14.021738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.021716    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:14.022252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.022229    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:14.023104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.023084    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:34:14.068517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.068491    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhjp\" (UniqueName: \"kubernetes.io/projected/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-kube-api-access-7bhjp\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.068719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.068697    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tmp-dir\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.068804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.068774    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-dshm\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.068804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.068799    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-model-cache\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.068907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.068815    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tls-certs\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.068907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.068867    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-home\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.069430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.069179    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tmp-dir\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.069430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.069243    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-model-cache\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.069689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.069667    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-home\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.072086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.072061    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-dshm\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.072309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.072291    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tls-certs\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.096047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.096022    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhjp\" (UniqueName: \"kubernetes.io/projected/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-kube-api-access-7bhjp\") pod \"distinct-w1-fc684b-kserve-599d8c66bc-htj6v\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.297156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.297066    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:14.610466 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:34:14.610435    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e8a4cf2_3f5f_4bff_960b_da8cfb664377.slice/crio-0d1c0d50ea69d53fa8f370c6cf5a258105056f75406683e25a89b5c2433f6887 WatchSource:0}: Error finding container 0d1c0d50ea69d53fa8f370c6cf5a258105056f75406683e25a89b5c2433f6887: Status 404 returned error can't find the container with id 0d1c0d50ea69d53fa8f370c6cf5a258105056f75406683e25a89b5c2433f6887
Oct 29 22:34:14.616051 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.616029    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"]
Oct 29 22:34:14.883264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:14.883179    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:15.023089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:15.023055    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:15.470715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:15.470677    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" event={"ID":"5e8a4cf2-3f5f-4bff-960b-da8cfb664377","Type":"ContainerStarted","Data":"afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597"}
Oct 29 22:34:15.470921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:15.470728    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" event={"ID":"5e8a4cf2-3f5f-4bff-960b-da8cfb664377","Type":"ContainerStarted","Data":"0d1c0d50ea69d53fa8f370c6cf5a258105056f75406683e25a89b5c2433f6887"}
Oct 29 22:34:15.514964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:15.514909    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" podStartSLOduration=2.514897468 podStartE2EDuration="2.514897468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:15.513533202 +0000 UTC m=+1197.538309352" watchObservedRunningTime="2026-10-29 22:34:15.514897468 +0000 UTC m=+1197.539673599"
Oct 29 22:34:16.032469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:16.032435    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"]
Oct 29 22:34:16.046439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:16.046409    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c39cbe7-95ctc"]
Oct 29 22:34:16.113637 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:16.113606    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171fd0ec-db45-49be-af25-3cd16e968bc2" path="/var/lib/kubelet/pods/171fd0ec-db45-49be-af25-3cd16e968bc2/volumes"
Oct 29 22:34:16.297555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:16.297469    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:16.712224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:16.712195    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:18.032528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:18.032495    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"]
Oct 29 22:34:18.046133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:18.046103    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441251fc-p7sb4"]
Oct 29 22:34:18.058933 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:34:18.058902    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:34:18.114058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:18.114032    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81017e47-c571-474e-a7ed-9aa0ca92a910" path="/var/lib/kubelet/pods/81017e47-c571-474e-a7ed-9aa0ca92a910/volumes"
Oct 29 22:34:18.296293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:18.296225    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:34:18.297440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:18.297419    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:34:18.317506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:18.317481    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:19.910848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:19.910811    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:20.918102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:20.918070    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:21.057841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:21.057814    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:21.256667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:21.256632    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:21.621140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:21.621039    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:23.242850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:23.242821    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:24.298581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:24.298547    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:24.311271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:24.311247    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:24.581185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:24.581105    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:34:24.857826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:24.857746    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:26.287914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:26.287884    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:27.298496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:27.298464    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:27.426459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:27.426425    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:27.592216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:27.592146    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:27.772849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:27.772819    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:28.151961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:28.151926    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:28.269707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:28.269670    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:29.073151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:29.073116    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"]
Oct 29 22:34:29.091726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:29.091695    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d216c885-k4twd"]
Oct 29 22:34:29.951844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:29.951632    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:30.122443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.122404    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00de079-cef7-43cb-ad71-c8109076fd07" path="/var/lib/kubelet/pods/d00de079-cef7-43cb-ad71-c8109076fd07/volumes"
Oct 29 22:34:30.349509 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.349434    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"]
Oct 29 22:34:30.359346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.357567    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:30.372350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.370986    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"]
Oct 29 22:34:30.390919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.384572    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:34:30.390919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.384712    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:34:30.400398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.400368    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:34:30.400863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.400843    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:34:30.539380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.536423    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e385da9b-776d6\" (UID: \"930ce2c1-91ec-47bd-8e53-f8489319b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:30.539380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.536570    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e385da9b-776d6\" (UID: \"930ce2c1-91ec-47bd-8e53-f8489319b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:30.539380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.536743    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp4h2\" (UniqueName: \"kubernetes.io/projected/930ce2c1-91ec-47bd-8e53-f8489319b202-kube-api-access-wp4h2\") pod \"maas-api-revoke-keys-e2e-migrate-e385da9b-776d6\" (UID: \"930ce2c1-91ec-47bd-8e53-f8489319b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:30.640399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.639719    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e385da9b-776d6\" (UID: \"930ce2c1-91ec-47bd-8e53-f8489319b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:30.640399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.639912    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp4h2\" (UniqueName: \"kubernetes.io/projected/930ce2c1-91ec-47bd-8e53-f8489319b202-kube-api-access-wp4h2\") pod \"maas-api-revoke-keys-e2e-migrate-e385da9b-776d6\" (UID: \"930ce2c1-91ec-47bd-8e53-f8489319b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:30.640399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.640147    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e385da9b-776d6\" (UID: \"930ce2c1-91ec-47bd-8e53-f8489319b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:30.642348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.641455    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e385da9b-776d6\" (UID: \"930ce2c1-91ec-47bd-8e53-f8489319b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:30.652284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.652257    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e385da9b-776d6\" (UID: \"930ce2c1-91ec-47bd-8e53-f8489319b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:30.663443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.663045    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp4h2\" (UniqueName: \"kubernetes.io/projected/930ce2c1-91ec-47bd-8e53-f8489319b202-kube-api-access-wp4h2\") pod \"maas-api-revoke-keys-e2e-migrate-e385da9b-776d6\" (UID: \"930ce2c1-91ec-47bd-8e53-f8489319b202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:30.749372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:30.748136    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:31.442837 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:31.442801    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"]
Oct 29 22:34:31.653029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:31.652945    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6" event={"ID":"930ce2c1-91ec-47bd-8e53-f8489319b202","Type":"ContainerStarted","Data":"4077f4becadf6686897c970e176216ff733a86125ad4edd6dcae44b9d7cb8107"}
Oct 29 22:34:31.976291 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:31.976247    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:32.663518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:32.663430    4860 generic.go:356] "Generic (PLEG): container finished" podID="930ce2c1-91ec-47bd-8e53-f8489319b202" containerID="8c0cd93050cc63b66c758f964fc6863a2d6339d107c03287062bdc801410484b" exitCode=0
Oct 29 22:34:32.663941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:32.663517    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6" event={"ID":"930ce2c1-91ec-47bd-8e53-f8489319b202","Type":"ContainerDied","Data":"8c0cd93050cc63b66c758f964fc6863a2d6339d107c03287062bdc801410484b"}
Oct 29 22:34:33.590250 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:33.590195    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:33.718127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:33.718085    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:33.846342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:33.846276    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:33.886353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:33.886316    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930ce2c1-91ec-47bd-8e53-f8489319b202-kube-api-access-wp4h2\" (UniqueName: \"kubernetes.io/projected/930ce2c1-91ec-47bd-8e53-f8489319b202-kube-api-access-wp4h2\") pod \"930ce2c1-91ec-47bd-8e53-f8489319b202\" (UID: \"930ce2c1-91ec-47bd-8e53-f8489319b202\") "
Oct 29 22:34:33.886490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:33.886378    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-openshift-ca\") pod \"930ce2c1-91ec-47bd-8e53-f8489319b202\" (UID: \"930ce2c1-91ec-47bd-8e53-f8489319b202\") "
Oct 29 22:34:33.886490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:33.886442    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-xks-ca\") pod \"930ce2c1-91ec-47bd-8e53-f8489319b202\" (UID: \"930ce2c1-91ec-47bd-8e53-f8489319b202\") "
Oct 29 22:34:33.886760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:33.886730    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-openshift-ca" pod "930ce2c1-91ec-47bd-8e53-f8489319b202" (UID: "930ce2c1-91ec-47bd-8e53-f8489319b202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:34:33.886897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:33.886882    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:33.888898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:33.888871    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-xks-ca" pod "930ce2c1-91ec-47bd-8e53-f8489319b202" (UID: "930ce2c1-91ec-47bd-8e53-f8489319b202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:34:33.888997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:33.888954    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930ce2c1-91ec-47bd-8e53-f8489319b202-kube-api-access-wp4h2" pod "930ce2c1-91ec-47bd-8e53-f8489319b202" (UID: "930ce2c1-91ec-47bd-8e53-f8489319b202"). InnerVolumeSpecName "kube-api-access-wp4h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:34:33.989392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:33.989364    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp4h2\" (UniqueName: \"kubernetes.io/projected/930ce2c1-91ec-47bd-8e53-f8489319b202-kube-api-access-wp4h2\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:33.989392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:33.989389    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930ce2c1-91ec-47bd-8e53-f8489319b202-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:34.682947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:34.682912    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"
Oct 29 22:34:34.683110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:34.682958    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6" event={"ID":"930ce2c1-91ec-47bd-8e53-f8489319b202","Type":"ContainerDied","Data":"4077f4becadf6686897c970e176216ff733a86125ad4edd6dcae44b9d7cb8107"}
Oct 29 22:34:34.683110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:34.683004    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4077f4becadf6686897c970e176216ff733a86125ad4edd6dcae44b9d7cb8107"
Oct 29 22:34:34.999431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:34.999359    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:35.329048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:35.328971    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:35.472600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:35.472565    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:35.655554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:35.655457    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:36.994457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:36.994424    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:38.655369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.655306    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:38.770539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.770505    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"]
Oct 29 22:34:38.773368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.773288    4860 cpu_manager.go:414] "containerMap: removing container" podUID="930ce2c1-91ec-47bd-8e53-f8489319b202" containerName="revoke-keys"
Oct 29 22:34:38.773368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.773315    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930ce2c1-91ec-47bd-8e53-f8489319b202" containerName="revoke-keys"
Oct 29 22:34:38.773908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.773878    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="930ce2c1-91ec-47bd-8e53-f8489319b202" containerName="revoke-keys"
Oct 29 22:34:38.783180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.783158    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"]
Oct 29 22:34:38.783295 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.783257    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:38.799568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.799539    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fc684b\"/\"distinct-2-w1-fc684b-kserve-self-signed-certs\""
Oct 29 22:34:38.829761 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.829735    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:38.879704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.879674    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-tmp-dir\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:38.879830 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.879760    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-model-cache\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:38.879920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.879901    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-home\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:38.880036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.880021    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/332d879d-e9f8-4f25-b6ef-3de840b331fc-tls-certs\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:38.880113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.880098    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-dshm\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:38.880188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.880172    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m76m\" (UniqueName: \"kubernetes.io/projected/332d879d-e9f8-4f25-b6ef-3de840b331fc-kube-api-access-5m76m\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:38.989260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.989111    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-dshm\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:38.990498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.990466    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m76m\" (UniqueName: \"kubernetes.io/projected/332d879d-e9f8-4f25-b6ef-3de840b331fc-kube-api-access-5m76m\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:38.991166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.991089    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-tmp-dir\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:38.997285 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.997260    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-model-cache\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:38.999818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.999794    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-home\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:39.000113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:39.000096    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/332d879d-e9f8-4f25-b6ef-3de840b331fc-tls-certs\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:39.001251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:39.001227    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-home\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:39.001469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.996984    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-tmp-dir\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:39.001584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:38.997735    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-model-cache\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:39.002503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:39.002483    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-dshm\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:39.019213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:39.019178    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/332d879d-e9f8-4f25-b6ef-3de840b331fc-tls-certs\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:39.023649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:39.023621    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m76m\" (UniqueName: \"kubernetes.io/projected/332d879d-e9f8-4f25-b6ef-3de840b331fc-kube-api-access-5m76m\") pod \"distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") " pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:39.100789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:39.100756    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:39.419916 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:34:39.419878    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod332d879d_e9f8_4f25_b6ef_3de840b331fc.slice/crio-8f7d67c73c06ff6d8c7080267a8700bf440860a43c3fd0d6f2eecb14c5a9543c WatchSource:0}: Error finding container 8f7d67c73c06ff6d8c7080267a8700bf440860a43c3fd0d6f2eecb14c5a9543c: Status 404 returned error can't find the container with id 8f7d67c73c06ff6d8c7080267a8700bf440860a43c3fd0d6f2eecb14c5a9543c
Oct 29 22:34:39.424263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:39.424241    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"]
Oct 29 22:34:39.788009 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:39.787966    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" event={"ID":"332d879d-e9f8-4f25-b6ef-3de840b331fc","Type":"ContainerStarted","Data":"a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd"}
Oct 29 22:34:39.788009 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:39.788009    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" event={"ID":"332d879d-e9f8-4f25-b6ef-3de840b331fc","Type":"ContainerStarted","Data":"8f7d67c73c06ff6d8c7080267a8700bf440860a43c3fd0d6f2eecb14c5a9543c"}
Oct 29 22:34:39.862544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:39.862493    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podStartSLOduration=1.8624794329999999 podStartE2EDuration="1.862479433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:39.856094671 +0000 UTC m=+1221.880870808" watchObservedRunningTime="2026-10-29 22:34:39.862479433 +0000 UTC m=+1221.887255564"
Oct 29 22:34:39.878178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:39.878127    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:40.013345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:40.006341    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:40.101147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:40.101060    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:40.613773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:40.613725    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:40.813979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:40.813949    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"]
Oct 29 22:34:40.849430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:40.849403    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"]
Oct 29 22:34:40.985546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:40.985512    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:41.109170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:41.109138    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:42.042264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:42.042229    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:42.210890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:42.210851    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:42.330662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:42.330585    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:42.636112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:42.636028    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:42.796262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:42.796229    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:42.808317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:42.808288    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"]
Oct 29 22:34:42.821176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:42.821147    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e385da9b-776d6"]
Oct 29 22:34:42.940511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:42.940310    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:43.058670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:43.058620    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:43.180143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:43.180110    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:44.113940 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:44.113906    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="930ce2c1-91ec-47bd-8e53-f8489319b202" path="/var/lib/kubelet/pods/930ce2c1-91ec-47bd-8e53-f8489319b202/volumes"
Oct 29 22:34:44.292526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:44.292497    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:44.696247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:44.696185    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:44.845901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:44.845621    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:45.399834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:45.399798    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:45.814597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:45.814556    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr" podUID="177e7be8-f825-44b8-a455-335a0039198a" containerName="payload-processing" containerID="cri-o://ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be" gracePeriod=25
Oct 29 22:34:45.850501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:45.850471    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw" podUID="e77feed5-cf09-42d0-9ab8-b9ff37753805" containerName="payload-pre-processing" containerID="cri-o://54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39" gracePeriod=25
Oct 29 22:34:46.202533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.195662    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"
Oct 29 22:34:46.277627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.274705    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177e7be8-f825-44b8-a455-335a0039198a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/177e7be8-f825-44b8-a455-335a0039198a-plugins-config-volume\") pod \"177e7be8-f825-44b8-a455-335a0039198a\" (UID: \"177e7be8-f825-44b8-a455-335a0039198a\") "
Oct 29 22:34:46.277627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.274952    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/177e7be8-f825-44b8-a455-335a0039198a-kube-api-access-cjrvz\" (UniqueName: \"kubernetes.io/projected/177e7be8-f825-44b8-a455-335a0039198a-kube-api-access-cjrvz\") pod \"177e7be8-f825-44b8-a455-335a0039198a\" (UID: \"177e7be8-f825-44b8-a455-335a0039198a\") "
Oct 29 22:34:46.277627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.275578    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177e7be8-f825-44b8-a455-335a0039198a-plugins-config-volume" pod "177e7be8-f825-44b8-a455-335a0039198a" (UID: "177e7be8-f825-44b8-a455-335a0039198a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:34:46.277627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.276380    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/177e7be8-f825-44b8-a455-335a0039198a-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:46.278656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.278628    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177e7be8-f825-44b8-a455-335a0039198a-kube-api-access-cjrvz" pod "177e7be8-f825-44b8-a455-335a0039198a" (UID: "177e7be8-f825-44b8-a455-335a0039198a"). InnerVolumeSpecName "kube-api-access-cjrvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:34:46.306371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.305928    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"
Oct 29 22:34:46.377695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.377605    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77feed5-cf09-42d0-9ab8-b9ff37753805-kube-api-access-2d65t\" (UniqueName: \"kubernetes.io/projected/e77feed5-cf09-42d0-9ab8-b9ff37753805-kube-api-access-2d65t\") pod \"e77feed5-cf09-42d0-9ab8-b9ff37753805\" (UID: \"e77feed5-cf09-42d0-9ab8-b9ff37753805\") "
Oct 29 22:34:46.377868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.377818    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77feed5-cf09-42d0-9ab8-b9ff37753805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77feed5-cf09-42d0-9ab8-b9ff37753805-plugins-config-volume\") pod \"e77feed5-cf09-42d0-9ab8-b9ff37753805\" (UID: \"e77feed5-cf09-42d0-9ab8-b9ff37753805\") "
Oct 29 22:34:46.378618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.378529    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77feed5-cf09-42d0-9ab8-b9ff37753805-plugins-config-volume" pod "e77feed5-cf09-42d0-9ab8-b9ff37753805" (UID: "e77feed5-cf09-42d0-9ab8-b9ff37753805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:34:46.380066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.380036    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjrvz\" (UniqueName: \"kubernetes.io/projected/177e7be8-f825-44b8-a455-335a0039198a-kube-api-access-cjrvz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:46.380066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.380060    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77feed5-cf09-42d0-9ab8-b9ff37753805-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:46.384824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.384800    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77feed5-cf09-42d0-9ab8-b9ff37753805-kube-api-access-2d65t" pod "e77feed5-cf09-42d0-9ab8-b9ff37753805" (UID: "e77feed5-cf09-42d0-9ab8-b9ff37753805"). InnerVolumeSpecName "kube-api-access-2d65t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:34:46.481045 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.481014    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d65t\" (UniqueName: \"kubernetes.io/projected/e77feed5-cf09-42d0-9ab8-b9ff37753805-kube-api-access-2d65t\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:46.542378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.542315    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:46.898577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.898535    4860 generic.go:356] "Generic (PLEG): container finished" podID="e77feed5-cf09-42d0-9ab8-b9ff37753805" containerID="54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39" exitCode=0
Oct 29 22:34:46.898771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.898605    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"
Oct 29 22:34:46.898771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.898622    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw" event={"ID":"e77feed5-cf09-42d0-9ab8-b9ff37753805","Type":"ContainerDied","Data":"54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39"}
Oct 29 22:34:46.898771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.898699    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw" event={"ID":"e77feed5-cf09-42d0-9ab8-b9ff37753805","Type":"ContainerDied","Data":"9bf453198bd0adcda4c4906535139163ffb45ff90a9f62b150f4f1156b0318af"}
Oct 29 22:34:46.898771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.898728    4860 scope.go:122] "RemoveContainer" containerID="54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39"
Oct 29 22:34:46.900805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.900780    4860 generic.go:356] "Generic (PLEG): container finished" podID="177e7be8-f825-44b8-a455-335a0039198a" containerID="ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be" exitCode=0
Oct 29 22:34:46.901065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.901032    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr" event={"ID":"177e7be8-f825-44b8-a455-335a0039198a","Type":"ContainerDied","Data":"ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be"}
Oct 29 22:34:46.901138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.901085    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr" event={"ID":"177e7be8-f825-44b8-a455-335a0039198a","Type":"ContainerDied","Data":"010ae2bd33880c5698271e98536f9bb9170382fa9d2edd698498f06d078567a5"}
Oct 29 22:34:46.901204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.901128    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"
Oct 29 22:34:46.918113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.918075    4860 scope.go:122] "RemoveContainer" containerID="54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39"
Oct 29 22:34:46.918572 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:34:46.918549    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39\": container with ID starting with 54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39 not found: ID does not exist" containerID="54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39"
Oct 29 22:34:46.918643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.918582    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39"} err="failed to get container status \"54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39\": rpc error: code = NotFound desc = could not find container \"54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39\": container with ID starting with 54f64c3dbd9c5928d2c6038fb536b1956238bdae08345b9aeeff4120992dbc39 not found: ID does not exist"
Oct 29 22:34:46.918643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.918603    4860 scope.go:122] "RemoveContainer" containerID="ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be"
Oct 29 22:34:46.932280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.932263    4860 scope.go:122] "RemoveContainer" containerID="ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be"
Oct 29 22:34:46.932582 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:34:46.932563    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be\": container with ID starting with ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be not found: ID does not exist" containerID="ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be"
Oct 29 22:34:46.932669 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.932588    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be"} err="failed to get container status \"ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be\": rpc error: code = NotFound desc = could not find container \"ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be\": container with ID starting with ee923b6e049fa85197ce371ddfe71c718f7eb727a81f67adc0a7712b22ebf7be not found: ID does not exist"
Oct 29 22:34:46.959489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.959462    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"]
Oct 29 22:34:46.977052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:46.977021    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e385da9b-64fcf8b6bc-zjwpw"]
Oct 29 22:34:47.018093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:47.018067    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"]
Oct 29 22:34:47.024640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:47.024616    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e385da9b-78d97b4574-h5mcr"]
Oct 29 22:34:48.057870 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:34:48.057831    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:34:48.080979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.080949    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:48.113636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.113607    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="177e7be8-f825-44b8-a455-335a0039198a" path="/var/lib/kubelet/pods/177e7be8-f825-44b8-a455-335a0039198a/volumes"
Oct 29 22:34:48.114033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.114019    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77feed5-cf09-42d0-9ab8-b9ff37753805" path="/var/lib/kubelet/pods/e77feed5-cf09-42d0-9ab8-b9ff37753805/volumes"
Oct 29 22:34:48.212541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.212509    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:48.348133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.348059    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:48.387068 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.387028    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"]
Oct 29 22:34:48.387737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.387719    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e77feed5-cf09-42d0-9ab8-b9ff37753805" containerName="payload-pre-processing"
Oct 29 22:34:48.387737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.387735    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77feed5-cf09-42d0-9ab8-b9ff37753805" containerName="payload-pre-processing"
Oct 29 22:34:48.387907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.387783    4860 cpu_manager.go:414] "containerMap: removing container" podUID="177e7be8-f825-44b8-a455-335a0039198a" containerName="payload-processing"
Oct 29 22:34:48.387907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.387788    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177e7be8-f825-44b8-a455-335a0039198a" containerName="payload-processing"
Oct 29 22:34:48.388131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.388107    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="177e7be8-f825-44b8-a455-335a0039198a" containerName="payload-processing"
Oct 29 22:34:48.388249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.388139    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77feed5-cf09-42d0-9ab8-b9ff37753805" containerName="payload-pre-processing"
Oct 29 22:34:48.393805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.393785    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:48.423160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.423131    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"]
Oct 29 22:34:48.424920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.424892    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:34:48.425063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.424984    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:34:48.425063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.425023    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:34:48.425769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.424874    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:34:48.506307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.506273    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc\" (UID: \"4c6744ad-3990-458a-88f3-90284b431b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:48.506594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.506576    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fst7\" (UniqueName: \"kubernetes.io/projected/4c6744ad-3990-458a-88f3-90284b431b6c-kube-api-access-4fst7\") pod \"maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc\" (UID: \"4c6744ad-3990-458a-88f3-90284b431b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:48.506644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.506633    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc\" (UID: \"4c6744ad-3990-458a-88f3-90284b431b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:48.608648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.608566    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fst7\" (UniqueName: \"kubernetes.io/projected/4c6744ad-3990-458a-88f3-90284b431b6c-kube-api-access-4fst7\") pod \"maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc\" (UID: \"4c6744ad-3990-458a-88f3-90284b431b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:48.608792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.608649    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc\" (UID: \"4c6744ad-3990-458a-88f3-90284b431b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:48.608792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.608717    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc\" (UID: \"4c6744ad-3990-458a-88f3-90284b431b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:48.609451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.609425    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc\" (UID: \"4c6744ad-3990-458a-88f3-90284b431b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:48.611110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.611086    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc\" (UID: \"4c6744ad-3990-458a-88f3-90284b431b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:48.632361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.632311    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fst7\" (UniqueName: \"kubernetes.io/projected/4c6744ad-3990-458a-88f3-90284b431b6c-kube-api-access-4fst7\") pod \"maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc\" (UID: \"4c6744ad-3990-458a-88f3-90284b431b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:48.720452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.720426    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:48.877944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:48.877867    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:49.022185 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:34:49.022156    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6744ad_3990_458a_88f3_90284b431b6c.slice/crio-e42f8938028dd178f8e2799e2593801f5acb9ae4e22aacc96a59ece83230f8ad WatchSource:0}: Error finding container e42f8938028dd178f8e2799e2593801f5acb9ae4e22aacc96a59ece83230f8ad: Status 404 returned error can't find the container with id e42f8938028dd178f8e2799e2593801f5acb9ae4e22aacc96a59ece83230f8ad
Oct 29 22:34:49.026945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:49.026915    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"]
Oct 29 22:34:49.101207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:49.101181    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:49.108239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:49.108216    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:49.114632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:49.114612    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:49.240099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:49.240074    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:49.939282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:49.939248    4860 generic.go:356] "Generic (PLEG): container finished" podID="4c6744ad-3990-458a-88f3-90284b431b6c" containerID="8d897306a3e6fe9483a5efc3a11c4a52b8285a321961f1bb9e4af0fbeb693ed3" exitCode=0
Oct 29 22:34:49.939459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:49.939355    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc" event={"ID":"4c6744ad-3990-458a-88f3-90284b431b6c","Type":"ContainerDied","Data":"8d897306a3e6fe9483a5efc3a11c4a52b8285a321961f1bb9e4af0fbeb693ed3"}
Oct 29 22:34:49.939459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:49.939415    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc" event={"ID":"4c6744ad-3990-458a-88f3-90284b431b6c","Type":"ContainerStarted","Data":"e42f8938028dd178f8e2799e2593801f5acb9ae4e22aacc96a59ece83230f8ad"}
Oct 29 22:34:49.951025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:49.951004    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:34:50.248366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:50.248300    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:50.861554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:50.861507    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:50.955771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:50.955713    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"]
Oct 29 22:34:50.960681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:50.960659    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"
Oct 29 22:34:50.978046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:50.977973    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:50.979408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:50.979384    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"]
Oct 29 22:34:50.981819 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:50.981795    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-aaa3c6\""
Oct 29 22:34:51.292938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.292897    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28zb\" (UniqueName: \"kubernetes.io/projected/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-kube-api-access-x28zb\") pod \"payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7\" (UID: \"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"
Oct 29 22:34:51.293359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.292994    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7\" (UID: \"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"
Oct 29 22:34:51.303594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.303573    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:51.334003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.333977    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"]
Oct 29 22:34:51.334722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.334702    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4c6744ad-3990-458a-88f3-90284b431b6c" containerName="revoke-keys"
Oct 29 22:34:51.334722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.334719    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6744ad-3990-458a-88f3-90284b431b6c" containerName="revoke-keys"
Oct 29 22:34:51.334899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.334825    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6744ad-3990-458a-88f3-90284b431b6c" containerName="revoke-keys"
Oct 29 22:34:51.338531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.338511    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.363375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.363287    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"]
Oct 29 22:34:51.394745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.394712    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6744ad-3990-458a-88f3-90284b431b6c-kube-api-access-4fst7\" (UniqueName: \"kubernetes.io/projected/4c6744ad-3990-458a-88f3-90284b431b6c-kube-api-access-4fst7\") pod \"4c6744ad-3990-458a-88f3-90284b431b6c\" (UID: \"4c6744ad-3990-458a-88f3-90284b431b6c\") "
Oct 29 22:34:51.394938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.394852    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-openshift-ca\") pod \"4c6744ad-3990-458a-88f3-90284b431b6c\" (UID: \"4c6744ad-3990-458a-88f3-90284b431b6c\") "
Oct 29 22:34:51.395013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.394974    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-xks-ca\") pod \"4c6744ad-3990-458a-88f3-90284b431b6c\" (UID: \"4c6744ad-3990-458a-88f3-90284b431b6c\") "
Oct 29 22:34:51.395525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.395273    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-openshift-ca" pod "4c6744ad-3990-458a-88f3-90284b431b6c" (UID: "4c6744ad-3990-458a-88f3-90284b431b6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:34:51.395525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.395420    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.395525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.395466    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.395769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.395568    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxs54\" (UniqueName: \"kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-kube-api-access-vxs54\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.395769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.395591    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.395769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.395682    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.395769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.395701    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.396046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.395792    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28zb\" (UniqueName: \"kubernetes.io/projected/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-kube-api-access-x28zb\") pod \"payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7\" (UID: \"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"
Oct 29 22:34:51.396046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.395815    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.396046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.395861    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.396046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.395892    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7\" (UID: \"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"
Oct 29 22:34:51.396046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.395923    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.396301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.396085    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:51.400455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.397313    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7\" (UID: \"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"
Oct 29 22:34:51.400455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.398044    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6744ad-3990-458a-88f3-90284b431b6c-kube-api-access-4fst7" pod "4c6744ad-3990-458a-88f3-90284b431b6c" (UID: "4c6744ad-3990-458a-88f3-90284b431b6c"). InnerVolumeSpecName "kube-api-access-4fst7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:34:51.400455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.398484    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-xks-ca" pod "4c6744ad-3990-458a-88f3-90284b431b6c" (UID: "4c6744ad-3990-458a-88f3-90284b431b6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:34:51.419558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.419501    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28zb\" (UniqueName: \"kubernetes.io/projected/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-kube-api-access-x28zb\") pod \"payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7\" (UID: \"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"
Oct 29 22:34:51.497612 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.497574    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxs54\" (UniqueName: \"kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-kube-api-access-vxs54\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.497787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.497616    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.497787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.497731    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.497787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.497760    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.497951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.497900    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.498014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.497957    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.498014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.497999    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.498123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.498109    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.498178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.498142    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.498265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.498243    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fst7\" (UniqueName: \"kubernetes.io/projected/4c6744ad-3990-458a-88f3-90284b431b6c-kube-api-access-4fst7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:51.498355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.498265    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6744ad-3990-458a-88f3-90284b431b6c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:34:51.498402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.498319    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.498437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.498411    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.498485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.498464    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.498529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.498465    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.498529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.498507    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.500281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.500258    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.500879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.500852    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.518868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.518817    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.519060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.519039    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxs54\" (UniqueName: \"kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-kube-api-access-vxs54\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.587126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.587105    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"
Oct 29 22:34:51.655664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.655639    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:51.893093 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:34:51.893063    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c5bd3e_fea2_497c_9e35_2d0c5a0077c9.slice/crio-6d19d8dac31eee50f7038d4a2162a7eca0c4d0a20af533aa3681517d258c0f78 WatchSource:0}: Error finding container 6d19d8dac31eee50f7038d4a2162a7eca0c4d0a20af533aa3681517d258c0f78: Status 404 returned error can't find the container with id 6d19d8dac31eee50f7038d4a2162a7eca0c4d0a20af533aa3681517d258c0f78
Oct 29 22:34:51.896607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.896579    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"]
Oct 29 22:34:51.963828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.963787    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7" event={"ID":"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9","Type":"ContainerStarted","Data":"6d19d8dac31eee50f7038d4a2162a7eca0c4d0a20af533aa3681517d258c0f78"}
Oct 29 22:34:51.965971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.965950    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"
Oct 29 22:34:51.966088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.965948    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc" event={"ID":"4c6744ad-3990-458a-88f3-90284b431b6c","Type":"ContainerDied","Data":"e42f8938028dd178f8e2799e2593801f5acb9ae4e22aacc96a59ece83230f8ad"}
Oct 29 22:34:51.966088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.966046    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e42f8938028dd178f8e2799e2593801f5acb9ae4e22aacc96a59ece83230f8ad"
Oct 29 22:34:51.988806 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:34:51.988776    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd673ec_820a_467b_9ccf_e2f5e4a0fbf4.slice/crio-ad42950bc3d6bd0e0e18838240e4eb7b54179c83040a63b07bbc397370aa4a04 WatchSource:0}: Error finding container ad42950bc3d6bd0e0e18838240e4eb7b54179c83040a63b07bbc397370aa4a04: Status 404 returned error can't find the container with id ad42950bc3d6bd0e0e18838240e4eb7b54179c83040a63b07bbc397370aa4a04
Oct 29 22:34:51.991290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.991268    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"]
Oct 29 22:34:51.991559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.991532    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:34:51.991604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.991592    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:34:51.991636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:51.991626    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:34:52.136757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:52.136706    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:52.980077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:52.980038    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7" event={"ID":"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9","Type":"ContainerStarted","Data":"726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801"}
Oct 29 22:34:52.980952 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:52.980131    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"
Oct 29 22:34:52.983402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:52.983319    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6" event={"ID":"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4","Type":"ContainerStarted","Data":"daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df"}
Oct 29 22:34:52.985482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:52.985447    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6" event={"ID":"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4","Type":"ContainerStarted","Data":"ad42950bc3d6bd0e0e18838240e4eb7b54179c83040a63b07bbc397370aa4a04"}
Oct 29 22:34:53.036108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:53.036059    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7" podStartSLOduration=3.036045743 podStartE2EDuration="3.036045743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:53.031105759 +0000 UTC m=+1235.055881891" watchObservedRunningTime="2026-10-29 22:34:53.036045743 +0000 UTC m=+1235.060821873"
Oct 29 22:34:53.095401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:53.095021    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6" podStartSLOduration=2.095001033 podStartE2EDuration="2.095001033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:53.085423301 +0000 UTC m=+1235.110199445" watchObservedRunningTime="2026-10-29 22:34:53.095001033 +0000 UTC m=+1235.119777166"
Oct 29 22:34:53.419491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:53.419405    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:53.539079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:53.539046    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:53.658298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:53.658264    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:53.660173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:53.660146    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:34:53.660253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:53.660199    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6" podUID="6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:34:53.750626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:53.750591    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:54.657785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:54.657750    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:34:54.658166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:54.657812    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6" podUID="6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:34:55.350414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:55.350361    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:55.464231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:55.464197    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:55.596349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:55.594261    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:55.658409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:55.658286    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:34:55.658409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:55.658386    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6" podUID="6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:34:56.658663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:56.658621    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:34:56.659086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:56.658691    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6" podUID="6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:34:56.799714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:56.799679    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"]
Oct 29 22:34:56.804504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:56.804476    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:34:56.817320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:56.817293    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"]
Oct 29 22:34:56.820890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:56.820864    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:34:56.821034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:56.821017    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:34:56.821111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:56.820914    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:34:56.821111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:56.820914    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:34:56.903455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:56.903420    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74vnd\" (UniqueName: \"kubernetes.io/projected/9e63f6bb-b80e-424a-b3bb-83043d00f544-kube-api-access-74vnd\") pod \"maas-api-revoke-sub-a73b5848-w8bpl\" (UID: \"9e63f6bb-b80e-424a-b3bb-83043d00f544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:34:56.903626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:56.903468    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a73b5848-w8bpl\" (UID: \"9e63f6bb-b80e-424a-b3bb-83043d00f544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:34:56.903680 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:56.903657    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a73b5848-w8bpl\" (UID: \"9e63f6bb-b80e-424a-b3bb-83043d00f544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:34:57.007208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:57.005319    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a73b5848-w8bpl\" (UID: \"9e63f6bb-b80e-424a-b3bb-83043d00f544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:34:57.007208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:57.005620    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vnd\" (UniqueName: \"kubernetes.io/projected/9e63f6bb-b80e-424a-b3bb-83043d00f544-kube-api-access-74vnd\") pod \"maas-api-revoke-sub-a73b5848-w8bpl\" (UID: \"9e63f6bb-b80e-424a-b3bb-83043d00f544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:34:57.007208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:57.005662    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a73b5848-w8bpl\" (UID: \"9e63f6bb-b80e-424a-b3bb-83043d00f544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:34:57.007208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:57.006694    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a73b5848-w8bpl\" (UID: \"9e63f6bb-b80e-424a-b3bb-83043d00f544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:34:57.013044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:57.013014    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a73b5848-w8bpl\" (UID: \"9e63f6bb-b80e-424a-b3bb-83043d00f544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:34:57.036004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:57.035974    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vnd\" (UniqueName: \"kubernetes.io/projected/9e63f6bb-b80e-424a-b3bb-83043d00f544-kube-api-access-74vnd\") pod \"maas-api-revoke-sub-a73b5848-w8bpl\" (UID: \"9e63f6bb-b80e-424a-b3bb-83043d00f544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:34:57.123296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:57.123267    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:34:57.293999 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:57.293845    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:57.454813 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:34:57.454785    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e63f6bb_b80e_424a_b3bb_83043d00f544.slice/crio-b0e3dbb22cf40b38a490fd88306cc8573a49300d95d39fb1f6ff7b832e0b5f98 WatchSource:0}: Error finding container b0e3dbb22cf40b38a490fd88306cc8573a49300d95d39fb1f6ff7b832e0b5f98: Status 404 returned error can't find the container with id b0e3dbb22cf40b38a490fd88306cc8573a49300d95d39fb1f6ff7b832e0b5f98
Oct 29 22:34:57.458513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:57.458490    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"]
Oct 29 22:34:57.658756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:57.658685    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:34:57.658756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:57.658743    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6" podUID="6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:34:58.049445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:58.048782    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl" event={"ID":"9e63f6bb-b80e-424a-b3bb-83043d00f544","Type":"ContainerStarted","Data":"09f9a8b44e814d82ee91473f2debe7e28499b08d1098a0409b5cf6bdb028a82d"}
Oct 29 22:34:58.049445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:58.048845    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl" event={"ID":"9e63f6bb-b80e-424a-b3bb-83043d00f544","Type":"ContainerStarted","Data":"b0e3dbb22cf40b38a490fd88306cc8573a49300d95d39fb1f6ff7b832e0b5f98"}
Oct 29 22:34:58.096686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:58.096627    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl" podStartSLOduration=2.096607844 podStartE2EDuration="2.096607844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:58.093991779 +0000 UTC m=+1240.118767913" watchObservedRunningTime="2026-10-29 22:34:58.096607844 +0000 UTC m=+1240.121383978"
Oct 29 22:34:58.658555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:58.658520    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:34:58.658682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:58.658584    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6" podUID="6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:34:58.872591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:58.872514    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:58.999353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:58.999309    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:59.145399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:59.143500    4860 generic.go:356] "Generic (PLEG): container finished" podID="9e63f6bb-b80e-424a-b3bb-83043d00f544" containerID="09f9a8b44e814d82ee91473f2debe7e28499b08d1098a0409b5cf6bdb028a82d" exitCode=0
Oct 29 22:34:59.145399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:59.143853    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl" event={"ID":"9e63f6bb-b80e-424a-b3bb-83043d00f544","Type":"ContainerDied","Data":"09f9a8b44e814d82ee91473f2debe7e28499b08d1098a0409b5cf6bdb028a82d"}
Oct 29 22:34:59.184576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:59.184527    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:34:59.986235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:59.986200    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:59.987792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:59.987768    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:34:59.988628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:34:59.988606    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:35:00.481487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:00.481462    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:35:00.604131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:00.604089    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-openshift-ca\") pod \"9e63f6bb-b80e-424a-b3bb-83043d00f544\" (UID: \"9e63f6bb-b80e-424a-b3bb-83043d00f544\") "
Oct 29 22:35:00.604340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:00.604146    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-xks-ca\") pod \"9e63f6bb-b80e-424a-b3bb-83043d00f544\" (UID: \"9e63f6bb-b80e-424a-b3bb-83043d00f544\") "
Oct 29 22:35:00.604340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:00.604226    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e63f6bb-b80e-424a-b3bb-83043d00f544-kube-api-access-74vnd\" (UniqueName: \"kubernetes.io/projected/9e63f6bb-b80e-424a-b3bb-83043d00f544-kube-api-access-74vnd\") pod \"9e63f6bb-b80e-424a-b3bb-83043d00f544\" (UID: \"9e63f6bb-b80e-424a-b3bb-83043d00f544\") "
Oct 29 22:35:00.604514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:00.604492    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-openshift-ca" pod "9e63f6bb-b80e-424a-b3bb-83043d00f544" (UID: "9e63f6bb-b80e-424a-b3bb-83043d00f544"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:35:00.605172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:00.605138    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:00.606751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:00.606724    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-xks-ca" pod "9e63f6bb-b80e-424a-b3bb-83043d00f544" (UID: "9e63f6bb-b80e-424a-b3bb-83043d00f544"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:35:00.607128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:00.607108    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e63f6bb-b80e-424a-b3bb-83043d00f544-kube-api-access-74vnd" pod "9e63f6bb-b80e-424a-b3bb-83043d00f544" (UID: "9e63f6bb-b80e-424a-b3bb-83043d00f544"). InnerVolumeSpecName "kube-api-access-74vnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:35:00.712819 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:00.712717    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e63f6bb-b80e-424a-b3bb-83043d00f544-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:00.712819 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:00.712761    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74vnd\" (UniqueName: \"kubernetes.io/projected/9e63f6bb-b80e-424a-b3bb-83043d00f544-kube-api-access-74vnd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:00.781206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:00.781020    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:01.206375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:01.205404    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"
Oct 29 22:35:01.206375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:01.205422    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl" event={"ID":"9e63f6bb-b80e-424a-b3bb-83043d00f544","Type":"ContainerDied","Data":"b0e3dbb22cf40b38a490fd88306cc8573a49300d95d39fb1f6ff7b832e0b5f98"}
Oct 29 22:35:01.206375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:01.205478    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0e3dbb22cf40b38a490fd88306cc8573a49300d95d39fb1f6ff7b832e0b5f98"
Oct 29 22:35:02.364751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:02.362899    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:02.784804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:02.784772    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:02.903371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:02.903318    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"]
Oct 29 22:35:02.918377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:02.917720    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c6e65e5-vxsnc"]
Oct 29 22:35:02.926039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:02.922031    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:03.235346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:03.234576    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"]
Oct 29 22:35:03.235346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:03.234813    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz" podUID="b808e4d2-a76c-4798-ae4d-74bbf0020fd5" containerName="istio-proxy" containerID="cri-o://984776f21e7d5e196baf8de5a629c317f813493b574d30d0808d13e1debf2147" gracePeriod=30
Oct 29 22:35:03.252743 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:03.245691    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:03.402347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:03.394959    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:04.006285 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:04.006251    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"
Oct 29 22:35:04.035867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:04.035836    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:04.121220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:04.121184    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6744ad-3990-458a-88f3-90284b431b6c" path="/var/lib/kubelet/pods/4c6744ad-3990-458a-88f3-90284b431b6c/volumes"
Oct 29 22:35:04.200193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:04.200155    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:04.322431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:04.322311    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:04.817856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:04.817827    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:04.934215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:04.934183    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:05.200815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:05.200781    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:05.337694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:05.337658    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:05.492441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:05.487730    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:06.718935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:06.718886    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:07.164320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.164286    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"]
Oct 29 22:35:07.164938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.164925    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9e63f6bb-b80e-424a-b3bb-83043d00f544" containerName="revoke-keys"
Oct 29 22:35:07.164987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.164940    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e63f6bb-b80e-424a-b3bb-83043d00f544" containerName="revoke-keys"
Oct 29 22:35:07.165089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.165080    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e63f6bb-b80e-424a-b3bb-83043d00f544" containerName="revoke-keys"
Oct 29 22:35:07.168801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.168783    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:07.192292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.192265    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"]
Oct 29 22:35:07.193183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.193156    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:35:07.193183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.193162    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:35:07.205543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.205524    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:35:07.205648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.205574    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:35:07.321816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.321784    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl4sl\" (UniqueName: \"kubernetes.io/projected/32c0fdd2-4ee7-4b7b-8432-cda376900576-kube-api-access-wl4sl\") pod \"maas-api-revoke-sub-db144c1c-tzcw5\" (UID: \"32c0fdd2-4ee7-4b7b-8432-cda376900576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:07.322010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.321991    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db144c1c-tzcw5\" (UID: \"32c0fdd2-4ee7-4b7b-8432-cda376900576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:07.322075 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.322060    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db144c1c-tzcw5\" (UID: \"32c0fdd2-4ee7-4b7b-8432-cda376900576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:07.423529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.423456    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db144c1c-tzcw5\" (UID: \"32c0fdd2-4ee7-4b7b-8432-cda376900576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:07.423529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.423528    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db144c1c-tzcw5\" (UID: \"32c0fdd2-4ee7-4b7b-8432-cda376900576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:07.423696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.423573    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl4sl\" (UniqueName: \"kubernetes.io/projected/32c0fdd2-4ee7-4b7b-8432-cda376900576-kube-api-access-wl4sl\") pod \"maas-api-revoke-sub-db144c1c-tzcw5\" (UID: \"32c0fdd2-4ee7-4b7b-8432-cda376900576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:07.424239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.424216    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db144c1c-tzcw5\" (UID: \"32c0fdd2-4ee7-4b7b-8432-cda376900576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:07.425996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.425974    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db144c1c-tzcw5\" (UID: \"32c0fdd2-4ee7-4b7b-8432-cda376900576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:07.449751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.449673    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl4sl\" (UniqueName: \"kubernetes.io/projected/32c0fdd2-4ee7-4b7b-8432-cda376900576-kube-api-access-wl4sl\") pod \"maas-api-revoke-sub-db144c1c-tzcw5\" (UID: \"32c0fdd2-4ee7-4b7b-8432-cda376900576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:07.486245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.486209    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:07.795925 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:35:07.795896    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c0fdd2_4ee7_4b7b_8432_cda376900576.slice/crio-b0da85f43b81f2db751b397c6e49d4851ec15ee1d6b042072588816badbaa9aa WatchSource:0}: Error finding container b0da85f43b81f2db751b397c6e49d4851ec15ee1d6b042072588816badbaa9aa: Status 404 returned error can't find the container with id b0da85f43b81f2db751b397c6e49d4851ec15ee1d6b042072588816badbaa9aa
Oct 29 22:35:07.803126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:07.803099    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"]
Oct 29 22:35:08.049939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:08.049846    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:08.328214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:08.328134    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:08.353300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:08.353264    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5" event={"ID":"32c0fdd2-4ee7-4b7b-8432-cda376900576","Type":"ContainerStarted","Data":"f3ad53237e76c533ff4efb4e5c3e8354c3b8b8205d6c05688cd976ddff024b99"}
Oct 29 22:35:08.353501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:08.353361    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5" event={"ID":"32c0fdd2-4ee7-4b7b-8432-cda376900576","Type":"ContainerStarted","Data":"b0da85f43b81f2db751b397c6e49d4851ec15ee1d6b042072588816badbaa9aa"}
Oct 29 22:35:08.403440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:08.403383    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5" podStartSLOduration=1.403365153 podStartE2EDuration="1.403365153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:08.396761446 +0000 UTC m=+1250.421537662" watchObservedRunningTime="2026-10-29 22:35:08.403365153 +0000 UTC m=+1250.428141282"
Oct 29 22:35:08.525593 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:08.525554    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:08.663397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:08.662675    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:09.366941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:09.366898    4860 generic.go:356] "Generic (PLEG): container finished" podID="32c0fdd2-4ee7-4b7b-8432-cda376900576" containerID="f3ad53237e76c533ff4efb4e5c3e8354c3b8b8205d6c05688cd976ddff024b99" exitCode=0
Oct 29 22:35:09.367399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:09.366996    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5" event={"ID":"32c0fdd2-4ee7-4b7b-8432-cda376900576","Type":"ContainerDied","Data":"f3ad53237e76c533ff4efb4e5c3e8354c3b8b8205d6c05688cd976ddff024b99"}
Oct 29 22:35:09.966364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:09.966303    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:10.384027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.383997    4860 generic.go:356] "Generic (PLEG): container finished" podID="b808e4d2-a76c-4798-ae4d-74bbf0020fd5" containerID="984776f21e7d5e196baf8de5a629c317f813493b574d30d0808d13e1debf2147" exitCode=0
Oct 29 22:35:10.384524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.384072    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz" event={"ID":"b808e4d2-a76c-4798-ae4d-74bbf0020fd5","Type":"ContainerDied","Data":"984776f21e7d5e196baf8de5a629c317f813493b574d30d0808d13e1debf2147"}
Oct 29 22:35:10.570160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.570133    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:35:10.605133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.605112    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:10.704350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.682758    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-socket\") pod \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") "
Oct 29 22:35:10.705469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.705440    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-certs\") pod \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") "
Oct 29 22:35:10.726427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.724785    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-token\" (UniqueName: \"kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-token\") pod \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") "
Oct 29 22:35:10.726729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.705358    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-socket" pod "b808e4d2-a76c-4798-ae4d-74bbf0020fd5" (UID: "b808e4d2-a76c-4798-ae4d-74bbf0020fd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:35:10.726729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.724171    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-certs" pod "b808e4d2-a76c-4798-ae4d-74bbf0020fd5" (UID: "b808e4d2-a76c-4798-ae4d-74bbf0020fd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:35:10.733307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.726668    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-envoy\") pod \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") "
Oct 29 22:35:10.733542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.733524    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-data\") pod \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") "
Oct 29 22:35:10.734223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.733898    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-xks-ca\") pod \"32c0fdd2-4ee7-4b7b-8432-cda376900576\" (UID: \"32c0fdd2-4ee7-4b7b-8432-cda376900576\") "
Oct 29 22:35:10.734437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.733829    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-data" pod "b808e4d2-a76c-4798-ae4d-74bbf0020fd5" (UID: "b808e4d2-a76c-4798-ae4d-74bbf0020fd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:35:10.739492 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.739463    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-openshift-ca\") pod \"32c0fdd2-4ee7-4b7b-8432-cda376900576\" (UID: \"32c0fdd2-4ee7-4b7b-8432-cda376900576\") "
Oct 29 22:35:10.739675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.739657    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-kube-api-access-lpm6v\" (UniqueName: \"kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-kube-api-access-lpm6v\") pod \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") "
Oct 29 22:35:10.739810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.739795    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istiod-ca-cert\") pod \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") "
Oct 29 22:35:10.739920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.739905    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32c0fdd2-4ee7-4b7b-8432-cda376900576-kube-api-access-wl4sl\" (UniqueName: \"kubernetes.io/projected/32c0fdd2-4ee7-4b7b-8432-cda376900576-kube-api-access-wl4sl\") pod \"32c0fdd2-4ee7-4b7b-8432-cda376900576\" (UID: \"32c0fdd2-4ee7-4b7b-8432-cda376900576\") "
Oct 29 22:35:10.740100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.740081    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-credential-socket\") pod \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") "
Oct 29 22:35:10.740345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.740311    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-podinfo\") pod \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\" (UID: \"b808e4d2-a76c-4798-ae4d-74bbf0020fd5\") "
Oct 29 22:35:10.741175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.741152    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:10.756729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.756706    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:10.756908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.756896    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:10.760522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.745670    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istiod-ca-cert" pod "b808e4d2-a76c-4798-ae4d-74bbf0020fd5" (UID: "b808e4d2-a76c-4798-ae4d-74bbf0020fd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:35:10.760522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.745912    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-openshift-ca" pod "32c0fdd2-4ee7-4b7b-8432-cda376900576" (UID: "32c0fdd2-4ee7-4b7b-8432-cda376900576"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:35:10.760522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.759404    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-credential-socket" pod "b808e4d2-a76c-4798-ae4d-74bbf0020fd5" (UID: "b808e4d2-a76c-4798-ae4d-74bbf0020fd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:35:10.792192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.791960    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-envoy" pod "b808e4d2-a76c-4798-ae4d-74bbf0020fd5" (UID: "b808e4d2-a76c-4798-ae4d-74bbf0020fd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:35:10.792192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.792059    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-podinfo" pod "b808e4d2-a76c-4798-ae4d-74bbf0020fd5" (UID: "b808e4d2-a76c-4798-ae4d-74bbf0020fd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:35:10.792641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.792593    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c0fdd2-4ee7-4b7b-8432-cda376900576-kube-api-access-wl4sl" pod "32c0fdd2-4ee7-4b7b-8432-cda376900576" (UID: "32c0fdd2-4ee7-4b7b-8432-cda376900576"). InnerVolumeSpecName "kube-api-access-wl4sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:35:10.792820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.792769    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-kube-api-access-lpm6v" pod "b808e4d2-a76c-4798-ae4d-74bbf0020fd5" (UID: "b808e4d2-a76c-4798-ae4d-74bbf0020fd5"). InnerVolumeSpecName "kube-api-access-lpm6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:35:10.800294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.795871    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-token" pod "b808e4d2-a76c-4798-ae4d-74bbf0020fd5" (UID: "b808e4d2-a76c-4798-ae4d-74bbf0020fd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:35:10.800294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.796054    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-xks-ca" pod "32c0fdd2-4ee7-4b7b-8432-cda376900576" (UID: "32c0fdd2-4ee7-4b7b-8432-cda376900576"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:35:10.859718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.859526    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:10.859718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.859564    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpm6v\" (UniqueName: \"kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-kube-api-access-lpm6v\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:10.859718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.859583    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:10.859718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.859598    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl4sl\" (UniqueName: \"kubernetes.io/projected/32c0fdd2-4ee7-4b7b-8432-cda376900576-kube-api-access-wl4sl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:10.859718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.859616    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:10.859718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.859630    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:10.859718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.859645    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:10.859718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.859657    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b808e4d2-a76c-4798-ae4d-74bbf0020fd5-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:10.859718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:10.859670    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32c0fdd2-4ee7-4b7b-8432-cda376900576-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:11.396063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:11.396030    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"
Oct 29 22:35:11.396487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:11.396061    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5" event={"ID":"32c0fdd2-4ee7-4b7b-8432-cda376900576","Type":"ContainerDied","Data":"b0da85f43b81f2db751b397c6e49d4851ec15ee1d6b042072588816badbaa9aa"}
Oct 29 22:35:11.396487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:11.396114    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0da85f43b81f2db751b397c6e49d4851ec15ee1d6b042072588816badbaa9aa"
Oct 29 22:35:11.398139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:11.398114    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"
Oct 29 22:35:11.398271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:11.398121    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz" event={"ID":"b808e4d2-a76c-4798-ae4d-74bbf0020fd5","Type":"ContainerDied","Data":"bc0dea5e0dd88afef894d5de67368707eb767ad53e5a8e5def647b4a522829f3"}
Oct 29 22:35:11.398354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:11.398251    4860 scope.go:122] "RemoveContainer" containerID="984776f21e7d5e196baf8de5a629c317f813493b574d30d0808d13e1debf2147"
Oct 29 22:35:11.413361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:11.413310    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:11.477536 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:11.477501    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"]
Oct 29 22:35:11.483056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:11.483030    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6c6e65e5-openshift-default-7586c4c97c-p8csz"]
Oct 29 22:35:11.677985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:11.677895    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:11.962557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:11.962517    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:12.115423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:12.115389    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b808e4d2-a76c-4798-ae4d-74bbf0020fd5" path="/var/lib/kubelet/pods/b808e4d2-a76c-4798-ae4d-74bbf0020fd5/volumes"
Oct 29 22:35:12.950875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:12.950840    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:13.065853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:13.065818    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:13.745750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:13.745717    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:14.620294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.620260    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:14.721668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.721639    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"]
Oct 29 22:35:14.722395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.722376    4860 cpu_manager.go:414] "containerMap: removing container" podUID="32c0fdd2-4ee7-4b7b-8432-cda376900576" containerName="revoke-keys"
Oct 29 22:35:14.722395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.722395    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32c0fdd2-4ee7-4b7b-8432-cda376900576" containerName="revoke-keys"
Oct 29 22:35:14.722534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.722426    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b808e4d2-a76c-4798-ae4d-74bbf0020fd5" containerName="istio-proxy"
Oct 29 22:35:14.722534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.722434    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b808e4d2-a76c-4798-ae4d-74bbf0020fd5" containerName="istio-proxy"
Oct 29 22:35:14.722627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.722614    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="32c0fdd2-4ee7-4b7b-8432-cda376900576" containerName="revoke-keys"
Oct 29 22:35:14.722663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.722631    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b808e4d2-a76c-4798-ae4d-74bbf0020fd5" containerName="istio-proxy"
Oct 29 22:35:14.728304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.728286    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"
Oct 29 22:35:14.740077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.740049    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:14.744494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.744469    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"]
Oct 29 22:35:14.755363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.750955    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1131a29d\""
Oct 29 22:35:14.790393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.790366    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"]
Oct 29 22:35:14.796267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.796248    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"
Oct 29 22:35:14.814864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.814845    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1131a29d-dockercfg-jt28l\""
Oct 29 22:35:14.818239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.818057    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"]
Oct 29 22:35:14.821854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.821823    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b3011f-d481-4687-851f-c170fe6ef09f-plugins-config-volume\") pod \"payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5\" (UID: \"02b3011f-d481-4687-851f-c170fe6ef09f\") " pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"
Oct 29 22:35:14.822133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.822092    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hx9\" (UniqueName: \"kubernetes.io/projected/02b3011f-d481-4687-851f-c170fe6ef09f-kube-api-access-w6hx9\") pod \"payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5\" (UID: \"02b3011f-d481-4687-851f-c170fe6ef09f\") " pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"
Oct 29 22:35:15.010347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:14.952291    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d3b90a-1eb7-4180-80f4-61b0e7647078-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2\" (UID: \"56d3b90a-1eb7-4180-80f4-61b0e7647078\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"
Oct 29 22:35:15.011274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.010898    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hx9\" (UniqueName: \"kubernetes.io/projected/02b3011f-d481-4687-851f-c170fe6ef09f-kube-api-access-w6hx9\") pod \"payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5\" (UID: \"02b3011f-d481-4687-851f-c170fe6ef09f\") " pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"
Oct 29 22:35:15.028374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.028340    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jckjd\" (UniqueName: \"kubernetes.io/projected/56d3b90a-1eb7-4180-80f4-61b0e7647078-kube-api-access-jckjd\") pod \"payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2\" (UID: \"56d3b90a-1eb7-4180-80f4-61b0e7647078\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"
Oct 29 22:35:15.029086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.029057    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b3011f-d481-4687-851f-c170fe6ef09f-plugins-config-volume\") pod \"payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5\" (UID: \"02b3011f-d481-4687-851f-c170fe6ef09f\") " pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"
Oct 29 22:35:15.035275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.035238    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b3011f-d481-4687-851f-c170fe6ef09f-plugins-config-volume\") pod \"payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5\" (UID: \"02b3011f-d481-4687-851f-c170fe6ef09f\") " pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"
Oct 29 22:35:15.047336 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.047297    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hx9\" (UniqueName: \"kubernetes.io/projected/02b3011f-d481-4687-851f-c170fe6ef09f-kube-api-access-w6hx9\") pod \"payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5\" (UID: \"02b3011f-d481-4687-851f-c170fe6ef09f\") " pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"
Oct 29 22:35:15.090200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.090168    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:15.133018 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.132988    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d3b90a-1eb7-4180-80f4-61b0e7647078-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2\" (UID: \"56d3b90a-1eb7-4180-80f4-61b0e7647078\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"
Oct 29 22:35:15.133185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.133162    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jckjd\" (UniqueName: \"kubernetes.io/projected/56d3b90a-1eb7-4180-80f4-61b0e7647078-kube-api-access-jckjd\") pod \"payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2\" (UID: \"56d3b90a-1eb7-4180-80f4-61b0e7647078\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"
Oct 29 22:35:15.133800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.133774    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d3b90a-1eb7-4180-80f4-61b0e7647078-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2\" (UID: \"56d3b90a-1eb7-4180-80f4-61b0e7647078\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"
Oct 29 22:35:15.155626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.155599    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jckjd\" (UniqueName: \"kubernetes.io/projected/56d3b90a-1eb7-4180-80f4-61b0e7647078-kube-api-access-jckjd\") pod \"payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2\" (UID: \"56d3b90a-1eb7-4180-80f4-61b0e7647078\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"
Oct 29 22:35:15.344533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.344435    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"
Oct 29 22:35:15.447698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.447661    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"
Oct 29 22:35:15.580210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.578592    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:15.789099 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:35:15.789070    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b3011f_d481_4687_851f_c170fe6ef09f.slice/crio-a5cca5deb7a81192d7b3957d23581c81326b1eeb8f892bb1b40c31f809408b51 WatchSource:0}: Error finding container a5cca5deb7a81192d7b3957d23581c81326b1eeb8f892bb1b40c31f809408b51: Status 404 returned error can't find the container with id a5cca5deb7a81192d7b3957d23581c81326b1eeb8f892bb1b40c31f809408b51
Oct 29 22:35:15.793042 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.793016    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"]
Oct 29 22:35:15.802579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:15.802553    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"]
Oct 29 22:35:15.803085 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:35:15.803063    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d3b90a_1eb7_4180_80f4_61b0e7647078.slice/crio-3d6213cd5d45e8d4e19b853af5871a43299773291a602f2eb67ce66a269c02fd WatchSource:0}: Error finding container 3d6213cd5d45e8d4e19b853af5871a43299773291a602f2eb67ce66a269c02fd: Status 404 returned error can't find the container with id 3d6213cd5d45e8d4e19b853af5871a43299773291a602f2eb67ce66a269c02fd
Oct 29 22:35:16.479954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:16.479922    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5" event={"ID":"02b3011f-d481-4687-851f-c170fe6ef09f","Type":"ContainerStarted","Data":"6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974"}
Oct 29 22:35:16.480167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:16.479975    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5" event={"ID":"02b3011f-d481-4687-851f-c170fe6ef09f","Type":"ContainerStarted","Data":"a5cca5deb7a81192d7b3957d23581c81326b1eeb8f892bb1b40c31f809408b51"}
Oct 29 22:35:16.480167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:16.480003    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"
Oct 29 22:35:16.481760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:16.481732    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2" event={"ID":"56d3b90a-1eb7-4180-80f4-61b0e7647078","Type":"ContainerStarted","Data":"c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe"}
Oct 29 22:35:16.481863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:16.481776    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2" event={"ID":"56d3b90a-1eb7-4180-80f4-61b0e7647078","Type":"ContainerStarted","Data":"3d6213cd5d45e8d4e19b853af5871a43299773291a602f2eb67ce66a269c02fd"}
Oct 29 22:35:16.481863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:16.481795    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"
Oct 29 22:35:16.522895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:16.522853    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5" podStartSLOduration=2.522838825 podStartE2EDuration="2.522838825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:16.520114258 +0000 UTC m=+1258.544890403" watchObservedRunningTime="2026-10-29 22:35:16.522838825 +0000 UTC m=+1258.547614955"
Oct 29 22:35:16.564982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:16.564935    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2" podStartSLOduration=2.564922784 podStartE2EDuration="2.564922784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:16.560427772 +0000 UTC m=+1258.585203926" watchObservedRunningTime="2026-10-29 22:35:16.564922784 +0000 UTC m=+1258.589698916"
Oct 29 22:35:17.459464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:17.459435    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:17.644523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:17.644486    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:18.057482 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:35:18.057444    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:35:18.452437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.452407    4860 scope.go:122] "RemoveContainer" containerID="14b04123a012203cdee7e2d85b927253def9a49d86626dd2a54a85a7a1649249"
Oct 29 22:35:18.463867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.463847    4860 scope.go:122] "RemoveContainer" containerID="4eb8898775461b38092c409da67b67b7533ea44018e31fb3dee6915f713f6804"
Oct 29 22:35:18.474096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.474081    4860 scope.go:122] "RemoveContainer" containerID="5de4998baee2c7d81acf9d8cdf1822c14ed32a2e87c11a5e950e3d317ef07132"
Oct 29 22:35:18.483861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.483840    4860 scope.go:122] "RemoveContainer" containerID="49ef0af0b6d622b116e625fe6cafacbcb5c7ae45727c4945a14faed3f4bb5563"
Oct 29 22:35:18.493468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.493452    4860 scope.go:122] "RemoveContainer" containerID="755f6f211f57fc67e8e720cbbf250a792157736bbf6fdfab103b66dfac5fde42"
Oct 29 22:35:18.503076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.503061    4860 scope.go:122] "RemoveContainer" containerID="d2589eb597c5c149d6698544a0113776ae5a470395d25bec6a0a0611c27b9c59"
Oct 29 22:35:18.516235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.516217    4860 scope.go:122] "RemoveContainer" containerID="b073e6640a70066345a86fcb60cbdb5b1be3bad513061257dafdef118b3dcfc9"
Oct 29 22:35:18.526856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.526840    4860 scope.go:122] "RemoveContainer" containerID="ecc1c2963d7edbad97c97719b37c787ed2324b57eb48fd0c8a44db0dab1cca91"
Oct 29 22:35:18.535885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.535870    4860 scope.go:122] "RemoveContainer" containerID="cacebe49b874d9effd21c97876c307684d9c5bb321639da666d922ec10831b3a"
Oct 29 22:35:18.545199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.545181    4860 scope.go:122] "RemoveContainer" containerID="48f44e124026fbf466cde28bea93bdbfe5dea84f5075b3f102b69a3600ae3bce"
Oct 29 22:35:18.554613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.554595    4860 scope.go:122] "RemoveContainer" containerID="4ae528fb19238c30386bac2d5d42808d26f87c0c97bf7faa9ab91932c42873cd"
Oct 29 22:35:18.564530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.564510    4860 scope.go:122] "RemoveContainer" containerID="812f1a0c4fc3c9fcb77e1139456d0e03423c54aae217b3d838753a8667386498"
Oct 29 22:35:18.997638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:18.997609    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:20.397632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:20.397599    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:20.663936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:20.663859    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:20.856594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:20.856563    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:20.974483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:20.974454    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:21.258204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.258124    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"]
Oct 29 22:35:21.266674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.266645    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.287909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.287866    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"]
Oct 29 22:35:21.291152 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.291131    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-db7419-openshift-default-dockercfg-s8vcb\""
Oct 29 22:35:21.336766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.336734    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:21.343162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.343136    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-certs\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.343274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.343243    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-credential-socket\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.343346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.343288    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a551643a-4108-4d0d-be08-6add34edffb1-istiod-ca-cert\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.343346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.343315    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-socket\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.343527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.343513    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-data\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.343562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.343553    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-istio-token\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.343671 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.343658    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a551643a-4108-4d0d-be08-6add34edffb1-istio-podinfo\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.343704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.343679    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gptr\" (UniqueName: \"kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-kube-api-access-6gptr\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.343753 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.343727    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-envoy\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.445241 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.445216    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-credential-socket\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.445614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.445256    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a551643a-4108-4d0d-be08-6add34edffb1-istiod-ca-cert\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.445614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.445284    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-socket\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.445614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.445438    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-data\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.445614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.445485    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-istio-token\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.445614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.445560    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a551643a-4108-4d0d-be08-6add34edffb1-istio-podinfo\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.445614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.445584    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gptr\" (UniqueName: \"kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-kube-api-access-6gptr\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.445957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.445640    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-envoy\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.445957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.445725    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-certs\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.445957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.445823    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-credential-socket\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.445957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.445827    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-socket\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.445957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.445836    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-data\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.446208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.446035    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-certs\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.446208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.446098    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a551643a-4108-4d0d-be08-6add34edffb1-istiod-ca-cert\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.448081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.448053    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-envoy\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.448286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.448269    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a551643a-4108-4d0d-be08-6add34edffb1-istio-podinfo\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.463032 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.463004    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:21.532800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.528576    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gptr\" (UniqueName: \"kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-kube-api-access-6gptr\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.532800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.532746    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-istio-token\") pod \"e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") " pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.588718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.588684    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:21.925433 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:35:21.925403    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda551643a_4108_4d0d_be08_6add34edffb1.slice/crio-0b8061cab42b3069f6f5b00ead23b623126a572e5fa7174102346574942ee2f0 WatchSource:0}: Error finding container 0b8061cab42b3069f6f5b00ead23b623126a572e5fa7174102346574942ee2f0: Status 404 returned error can't find the container with id 0b8061cab42b3069f6f5b00ead23b623126a572e5fa7174102346574942ee2f0
Oct 29 22:35:21.928126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.928092    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:21.928207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.928158    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:21.928207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.928189    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:21.932579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:21.932551    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"]
Oct 29 22:35:22.262885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:22.262845    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:22.585376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:22.585276    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" event={"ID":"a551643a-4108-4d0d-be08-6add34edffb1","Type":"ContainerStarted","Data":"3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386"}
Oct 29 22:35:22.585724 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:22.585377    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" event={"ID":"a551643a-4108-4d0d-be08-6add34edffb1","Type":"ContainerStarted","Data":"0b8061cab42b3069f6f5b00ead23b623126a572e5fa7174102346574942ee2f0"}
Oct 29 22:35:22.589600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:22.589584    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:22.649585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:22.649531    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" podStartSLOduration=1.649518809 podStartE2EDuration="1.649518809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:22.644595387 +0000 UTC m=+1264.669371528" watchObservedRunningTime="2026-10-29 22:35:22.649518809 +0000 UTC m=+1264.674294939"
Oct 29 22:35:23.219118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:23.219085    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:23.359451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:23.359394    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:23.595127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:23.595049    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:23.868257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:23.868167    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:24.543707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:24.543676    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:24.605528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:24.605498    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:24.606508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:24.606482    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:35:24.999742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:24.999715    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:25.188156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:25.188128    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:25.346260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:25.346174    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:25.654891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:25.654818    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:25.787874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:25.787834    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:25.922838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:25.922711    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:26.911286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:26.911253    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:27.032597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.032559    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:27.055935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.055906    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"]
Oct 29 22:35:27.062405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.062384    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"
Oct 29 22:35:27.065366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.065304    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc\" (UID: \"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"
Oct 29 22:35:27.065523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.065503    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqvh4\" (UniqueName: \"kubernetes.io/projected/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-kube-api-access-cqvh4\") pod \"payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc\" (UID: \"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"
Oct 29 22:35:27.085608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.085583    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"]
Oct 29 22:35:27.090055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.090032    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-db7419-dockercfg-d88g5\""
Oct 29 22:35:27.090155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.090091    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-db7419\""
Oct 29 22:35:27.206410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.206377    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"]
Oct 29 22:35:27.222798 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.222589    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqvh4\" (UniqueName: \"kubernetes.io/projected/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-kube-api-access-cqvh4\") pod \"payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc\" (UID: \"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"
Oct 29 22:35:27.223408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.223371    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc\" (UID: \"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"
Oct 29 22:35:27.224608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.224556    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc\" (UID: \"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"
Oct 29 22:35:27.230724 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.230703    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"
Oct 29 22:35:27.319919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.319882    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqvh4\" (UniqueName: \"kubernetes.io/projected/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-kube-api-access-cqvh4\") pod \"payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc\" (UID: \"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"
Oct 29 22:35:27.320102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.320085    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"]
Oct 29 22:35:27.327307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.327270    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2\" (UID: \"a29d3368-acf8-4eca-a0aa-e5b11d83f05a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"
Oct 29 22:35:27.327693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.327669    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nv9b\" (UniqueName: \"kubernetes.io/projected/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-kube-api-access-4nv9b\") pod \"payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2\" (UID: \"a29d3368-acf8-4eca-a0aa-e5b11d83f05a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"
Oct 29 22:35:27.377436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.377366    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"
Oct 29 22:35:27.429688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.429653    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2\" (UID: \"a29d3368-acf8-4eca-a0aa-e5b11d83f05a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"
Oct 29 22:35:27.429885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.429861    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nv9b\" (UniqueName: \"kubernetes.io/projected/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-kube-api-access-4nv9b\") pod \"payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2\" (UID: \"a29d3368-acf8-4eca-a0aa-e5b11d83f05a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"
Oct 29 22:35:27.431029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.430992    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2\" (UID: \"a29d3368-acf8-4eca-a0aa-e5b11d83f05a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"
Oct 29 22:35:27.456255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.456229    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nv9b\" (UniqueName: \"kubernetes.io/projected/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-kube-api-access-4nv9b\") pod \"payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2\" (UID: \"a29d3368-acf8-4eca-a0aa-e5b11d83f05a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"
Oct 29 22:35:27.509032 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.508622    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"
Oct 29 22:35:27.510354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.509924    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"
Oct 29 22:35:27.619665 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.619632    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"
Oct 29 22:35:27.700855 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:35:27.700811    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72b6fe8_c4f8_4ab3_bfa6_ef4edbf77c91.slice/crio-95b4a38cd539265551ba8dca1eaa08f576c70d8e6670395e2249ece8f71b1010 WatchSource:0}: Error finding container 95b4a38cd539265551ba8dca1eaa08f576c70d8e6670395e2249ece8f71b1010: Status 404 returned error can't find the container with id 95b4a38cd539265551ba8dca1eaa08f576c70d8e6670395e2249ece8f71b1010
Oct 29 22:35:27.700986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.700832    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"]
Oct 29 22:35:27.759029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.758982    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:27.933315 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:35:27.933287    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29d3368_acf8_4eca_a0aa_e5b11d83f05a.slice/crio-caad27b8c7ce0b792b8ec32aefa86c652a7c7b8b249ee321ba996a22294c7508 WatchSource:0}: Error finding container caad27b8c7ce0b792b8ec32aefa86c652a7c7b8b249ee321ba996a22294c7508: Status 404 returned error can't find the container with id caad27b8c7ce0b792b8ec32aefa86c652a7c7b8b249ee321ba996a22294c7508
Oct 29 22:35:27.937596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:27.937572    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"]
Oct 29 22:35:28.649098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:28.649059    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2" event={"ID":"a29d3368-acf8-4eca-a0aa-e5b11d83f05a","Type":"ContainerStarted","Data":"38829dc4a397852c4629be992ef281191313ae8298cebae6695460554384594f"}
Oct 29 22:35:28.649284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:28.649122    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2" event={"ID":"a29d3368-acf8-4eca-a0aa-e5b11d83f05a","Type":"ContainerStarted","Data":"caad27b8c7ce0b792b8ec32aefa86c652a7c7b8b249ee321ba996a22294c7508"}
Oct 29 22:35:28.649284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:28.649147    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"
Oct 29 22:35:28.650808 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:28.650780    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc" event={"ID":"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91","Type":"ContainerStarted","Data":"c7042a40ea8f1d51367de9b53c7638f05b0e61cf8e143f28c4381f79b13ec0e6"}
Oct 29 22:35:28.650949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:28.650826    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc" event={"ID":"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91","Type":"ContainerStarted","Data":"95b4a38cd539265551ba8dca1eaa08f576c70d8e6670395e2249ece8f71b1010"}
Oct 29 22:35:28.650949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:28.650848    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"
Oct 29 22:35:28.693973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:28.693932    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2" podStartSLOduration=1.6939207440000001 podStartE2EDuration="1.693920744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:28.690910431 +0000 UTC m=+1270.715686564" watchObservedRunningTime="2026-10-29 22:35:28.693920744 +0000 UTC m=+1270.718696878"
Oct 29 22:35:28.733046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:28.733007    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc" podStartSLOduration=1.732997763 podStartE2EDuration="1.732997763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:28.732242996 +0000 UTC m=+1270.757019127" watchObservedRunningTime="2026-10-29 22:35:28.732997763 +0000 UTC m=+1270.757773894"
Oct 29 22:35:29.121744 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.121712    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:29.251551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.251507    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:29.443489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.443456    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"]
Oct 29 22:35:29.448342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.448311    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:29.465129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.465107    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:35:29.469278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.469252    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"]
Oct 29 22:35:29.474416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.474391    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:35:29.474526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.474487    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:35:29.478919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.478897    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:35:29.558704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.558670    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5720f895-nvhnf\" (UID: \"4aa59d6b-9bec-433d-bf28-19992dc89f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:29.558865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.558719    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5720f895-nvhnf\" (UID: \"4aa59d6b-9bec-433d-bf28-19992dc89f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:29.558865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.558787    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx78b\" (UniqueName: \"kubernetes.io/projected/4aa59d6b-9bec-433d-bf28-19992dc89f36-kube-api-access-kx78b\") pod \"maas-api-revoke-sub-5720f895-nvhnf\" (UID: \"4aa59d6b-9bec-433d-bf28-19992dc89f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:29.636071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.636042    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:29.660668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.660636    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5720f895-nvhnf\" (UID: \"4aa59d6b-9bec-433d-bf28-19992dc89f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:29.660835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.660680    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5720f895-nvhnf\" (UID: \"4aa59d6b-9bec-433d-bf28-19992dc89f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:29.660835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.660736    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx78b\" (UniqueName: \"kubernetes.io/projected/4aa59d6b-9bec-433d-bf28-19992dc89f36-kube-api-access-kx78b\") pod \"maas-api-revoke-sub-5720f895-nvhnf\" (UID: \"4aa59d6b-9bec-433d-bf28-19992dc89f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:29.661450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.661429    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5720f895-nvhnf\" (UID: \"4aa59d6b-9bec-433d-bf28-19992dc89f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:29.662994 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.662977    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5720f895-nvhnf\" (UID: \"4aa59d6b-9bec-433d-bf28-19992dc89f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:29.682820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.682792    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx78b\" (UniqueName: \"kubernetes.io/projected/4aa59d6b-9bec-433d-bf28-19992dc89f36-kube-api-access-kx78b\") pod \"maas-api-revoke-sub-5720f895-nvhnf\" (UID: \"4aa59d6b-9bec-433d-bf28-19992dc89f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:29.763346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.763242    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:29.770018 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:29.768102    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:30.129707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:30.129675    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"]
Oct 29 22:35:30.673804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:30.673699    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf" event={"ID":"4aa59d6b-9bec-433d-bf28-19992dc89f36","Type":"ContainerStarted","Data":"c08b0189edc4c326a2665959a08f241f3808de327c8451db562ad2e07c4d219f"}
Oct 29 22:35:30.673804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:30.673775    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf" event={"ID":"4aa59d6b-9bec-433d-bf28-19992dc89f36","Type":"ContainerStarted","Data":"a6ccdc5db5fa07f91406bec9654629394a5e4e1e06b6abe687db64131c88c723"}
Oct 29 22:35:30.723585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:30.723517    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf" podStartSLOduration=1.723498287 podStartE2EDuration="1.723498287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:30.716165924 +0000 UTC m=+1272.740942056" watchObservedRunningTime="2026-10-29 22:35:30.723498287 +0000 UTC m=+1272.748274418"
Oct 29 22:35:30.901915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:30.901870    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:31.045818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.045775    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:31.050992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.050966    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"]
Oct 29 22:35:31.056415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.056396    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"
Oct 29 22:35:31.072599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.072570    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-3bd0f6-dockercfg-9tdvh\""
Oct 29 22:35:31.072599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.072591    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-3bd0f6\""
Oct 29 22:35:31.074846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.074824    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"]
Oct 29 22:35:31.334412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.333257    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"]
Oct 29 22:35:31.339538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.339509    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdkpj\" (UniqueName: \"kubernetes.io/projected/335d830d-7cdd-4527-a215-6f247fbf11b8-kube-api-access-mdkpj\") pod \"payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6\" (UID: \"335d830d-7cdd-4527-a215-6f247fbf11b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"
Oct 29 22:35:31.339810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.339792    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335d830d-7cdd-4527-a215-6f247fbf11b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6\" (UID: \"335d830d-7cdd-4527-a215-6f247fbf11b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"
Oct 29 22:35:31.340948 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.340271    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"
Oct 29 22:35:31.357157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.357116    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"]
Oct 29 22:35:31.441900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.441866    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vmtk\" (UniqueName: \"kubernetes.io/projected/c7208a84-7581-40bd-8b3e-f8d399feabb0-kube-api-access-6vmtk\") pod \"payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t\" (UID: \"c7208a84-7581-40bd-8b3e-f8d399feabb0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"
Oct 29 22:35:31.442076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.442020    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdkpj\" (UniqueName: \"kubernetes.io/projected/335d830d-7cdd-4527-a215-6f247fbf11b8-kube-api-access-mdkpj\") pod \"payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6\" (UID: \"335d830d-7cdd-4527-a215-6f247fbf11b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"
Oct 29 22:35:31.442151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.442090    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335d830d-7cdd-4527-a215-6f247fbf11b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6\" (UID: \"335d830d-7cdd-4527-a215-6f247fbf11b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"
Oct 29 22:35:31.442238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.442220    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7208a84-7581-40bd-8b3e-f8d399feabb0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t\" (UID: \"c7208a84-7581-40bd-8b3e-f8d399feabb0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"
Oct 29 22:35:31.442850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.442827    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335d830d-7cdd-4527-a215-6f247fbf11b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6\" (UID: \"335d830d-7cdd-4527-a215-6f247fbf11b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"
Oct 29 22:35:31.463398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.463366    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdkpj\" (UniqueName: \"kubernetes.io/projected/335d830d-7cdd-4527-a215-6f247fbf11b8-kube-api-access-mdkpj\") pod \"payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6\" (UID: \"335d830d-7cdd-4527-a215-6f247fbf11b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"
Oct 29 22:35:31.543773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.543629    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7208a84-7581-40bd-8b3e-f8d399feabb0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t\" (UID: \"c7208a84-7581-40bd-8b3e-f8d399feabb0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"
Oct 29 22:35:31.545738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.544114    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vmtk\" (UniqueName: \"kubernetes.io/projected/c7208a84-7581-40bd-8b3e-f8d399feabb0-kube-api-access-6vmtk\") pod \"payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t\" (UID: \"c7208a84-7581-40bd-8b3e-f8d399feabb0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"
Oct 29 22:35:31.545738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.545690    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7208a84-7581-40bd-8b3e-f8d399feabb0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t\" (UID: \"c7208a84-7581-40bd-8b3e-f8d399feabb0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"
Oct 29 22:35:31.568126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.568062    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vmtk\" (UniqueName: \"kubernetes.io/projected/c7208a84-7581-40bd-8b3e-f8d399feabb0-kube-api-access-6vmtk\") pod \"payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t\" (UID: \"c7208a84-7581-40bd-8b3e-f8d399feabb0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"
Oct 29 22:35:31.647424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.646426    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:31.671725 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.671686    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"
Oct 29 22:35:31.676478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.676449    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"
Oct 29 22:35:31.704673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.704587    4860 generic.go:356] "Generic (PLEG): container finished" podID="4aa59d6b-9bec-433d-bf28-19992dc89f36" containerID="c08b0189edc4c326a2665959a08f241f3808de327c8451db562ad2e07c4d219f" exitCode=0
Oct 29 22:35:31.704864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:31.704835    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf" event={"ID":"4aa59d6b-9bec-433d-bf28-19992dc89f36","Type":"ContainerDied","Data":"c08b0189edc4c326a2665959a08f241f3808de327c8451db562ad2e07c4d219f"}
Oct 29 22:35:32.034003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.033972    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"]
Oct 29 22:35:32.040140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.040111    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8959cc98-fxftk"]
Oct 29 22:35:32.069505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.069129    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"]
Oct 29 22:35:32.081816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.081786    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"]
Oct 29 22:35:32.116367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.116340    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f69917-9bbe-452b-8b61-35bd89d82705" path="/var/lib/kubelet/pods/49f69917-9bbe-452b-8b61-35bd89d82705/volumes"
Oct 29 22:35:32.721785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.721753    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t" event={"ID":"c7208a84-7581-40bd-8b3e-f8d399feabb0","Type":"ContainerStarted","Data":"bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654"}
Oct 29 22:35:32.722230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.721816    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t" event={"ID":"c7208a84-7581-40bd-8b3e-f8d399feabb0","Type":"ContainerStarted","Data":"c43f65813ff09552c756fa08dc4ad8edeecf2b13aef837e422df3b5143f7706d"}
Oct 29 22:35:32.722230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.721846    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"
Oct 29 22:35:32.724227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.724124    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6" event={"ID":"335d830d-7cdd-4527-a215-6f247fbf11b8","Type":"ContainerStarted","Data":"b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a"}
Oct 29 22:35:32.724227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.724170    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6" event={"ID":"335d830d-7cdd-4527-a215-6f247fbf11b8","Type":"ContainerStarted","Data":"ff7e7ba7f820abf0991c8554517c936939e16c5a47bc0f9a0cb2b1e4d898fa16"}
Oct 29 22:35:32.773570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.773509    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t" podStartSLOduration=1.773489631 podStartE2EDuration="1.773489631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:32.76629507 +0000 UTC m=+1274.791071211" watchObservedRunningTime="2026-10-29 22:35:32.773489631 +0000 UTC m=+1274.798265763"
Oct 29 22:35:32.813398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.813313    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6" podStartSLOduration=1.813297349 podStartE2EDuration="1.813297349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:32.809770847 +0000 UTC m=+1274.834546979" watchObservedRunningTime="2026-10-29 22:35:32.813297349 +0000 UTC m=+1274.838073479"
Oct 29 22:35:32.917148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.917124    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:32.978364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.978256    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa59d6b-9bec-433d-bf28-19992dc89f36-kube-api-access-kx78b\" (UniqueName: \"kubernetes.io/projected/4aa59d6b-9bec-433d-bf28-19992dc89f36-kube-api-access-kx78b\") pod \"4aa59d6b-9bec-433d-bf28-19992dc89f36\" (UID: \"4aa59d6b-9bec-433d-bf28-19992dc89f36\") "
Oct 29 22:35:32.978546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.978362    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-openshift-ca\") pod \"4aa59d6b-9bec-433d-bf28-19992dc89f36\" (UID: \"4aa59d6b-9bec-433d-bf28-19992dc89f36\") "
Oct 29 22:35:32.978546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.978496    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-xks-ca\") pod \"4aa59d6b-9bec-433d-bf28-19992dc89f36\" (UID: \"4aa59d6b-9bec-433d-bf28-19992dc89f36\") "
Oct 29 22:35:32.979374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.979288    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-openshift-ca" pod "4aa59d6b-9bec-433d-bf28-19992dc89f36" (UID: "4aa59d6b-9bec-433d-bf28-19992dc89f36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:35:32.992764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.992725    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa59d6b-9bec-433d-bf28-19992dc89f36-kube-api-access-kx78b" pod "4aa59d6b-9bec-433d-bf28-19992dc89f36" (UID: "4aa59d6b-9bec-433d-bf28-19992dc89f36"). InnerVolumeSpecName "kube-api-access-kx78b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:35:32.994297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:32.994268    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-xks-ca" pod "4aa59d6b-9bec-433d-bf28-19992dc89f36" (UID: "4aa59d6b-9bec-433d-bf28-19992dc89f36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:35:33.080503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:33.080428    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx78b\" (UniqueName: \"kubernetes.io/projected/4aa59d6b-9bec-433d-bf28-19992dc89f36-kube-api-access-kx78b\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:33.080714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:33.080694    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:33.080802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:33.080718    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa59d6b-9bec-433d-bf28-19992dc89f36-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:35:33.335288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:33.335196    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:33.734073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:33.734045    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"
Oct 29 22:35:33.734073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:33.734062    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf" event={"ID":"4aa59d6b-9bec-433d-bf28-19992dc89f36","Type":"ContainerDied","Data":"a6ccdc5db5fa07f91406bec9654629394a5e4e1e06b6abe687db64131c88c723"}
Oct 29 22:35:33.734577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:33.734110    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6ccdc5db5fa07f91406bec9654629394a5e4e1e06b6abe687db64131c88c723"
Oct 29 22:35:33.734577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:33.734424    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"
Oct 29 22:35:34.935381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:34.935320    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:36.468559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.468530    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"]
Oct 29 22:35:36.469220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.469205    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4aa59d6b-9bec-433d-bf28-19992dc89f36" containerName="revoke-keys"
Oct 29 22:35:36.469220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.469219    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa59d6b-9bec-433d-bf28-19992dc89f36" containerName="revoke-keys"
Oct 29 22:35:36.469412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.469401    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa59d6b-9bec-433d-bf28-19992dc89f36" containerName="revoke-keys"
Oct 29 22:35:36.473664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.473630    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.504251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.504228    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"]
Oct 29 22:35:36.514677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.514652    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"]
Oct 29 22:35:36.520793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.520769    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.525973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.525378    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.525973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.525563    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.525973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.525675    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.525973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.525870    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.525973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.525905    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.529134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.529109    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.529246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.529207    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.529364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.529307    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.529446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.529363    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmln\" (UniqueName: \"kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-kube-api-access-wvmln\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.533383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.533363    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"]
Oct 29 22:35:36.565015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.564976    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:36.630698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.630665    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.630845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.630717    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.630845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.630750    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.630845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.630772    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.630845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.630789    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.630845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.630807    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.631031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.630990    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.631070 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.631050    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.631119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.631077    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srbxs\" (UniqueName: \"kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-kube-api-access-srbxs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.631201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.631169    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.631343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.631199    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.631343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.631236    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.631343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.631281    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.631533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.631361    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.631533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.631403    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.633387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.631474    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.633515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.631487    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.633515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.633461    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmln\" (UniqueName: \"kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-kube-api-access-wvmln\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.633636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.633534    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.633636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.633572    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc12952-6dd5-4269-aa22-55b256dddbc1-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.633754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.633636    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.633754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.631816    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.633754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.631777    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.633754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.632426    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.634518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.634491    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.652072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.652047    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.653923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.653897    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmln\" (UniqueName: \"kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-kube-api-access-wvmln\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.735210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.735113    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.735210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.735179    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.735523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.735227    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srbxs\" (UniqueName: \"kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-kube-api-access-srbxs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.735523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.735383    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.735523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.735514    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.735702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.735539    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc12952-6dd5-4269-aa22-55b256dddbc1-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.735702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.735608    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-credential-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.735702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.735617    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.735702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.735657    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-certs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.735907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.735704    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-data\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.735907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.735733    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.735907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.735775    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.736058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.736032    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-socket\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.736224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.736205    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc12952-6dd5-4269-aa22-55b256dddbc1-istiod-ca-cert\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.737888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.737866    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-envoy\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.738129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.738110    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-podinfo\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.758705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.758683    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srbxs\" (UniqueName: \"kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-kube-api-access-srbxs\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.762136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.762112    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-token\") pod \"e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") " pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:36.788743 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.788714    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:36.842601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:36.842572    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:37.112439 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:35:37.112412    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3ea6e5_4dea_45ea_905e_b8091d29b209.slice/crio-dafc98cc33fd623604f07d603173c3e0cedc96962d83db75df83384c6cd10087 WatchSource:0}: Error finding container dafc98cc33fd623604f07d603173c3e0cedc96962d83db75df83384c6cd10087: Status 404 returned error can't find the container with id dafc98cc33fd623604f07d603173c3e0cedc96962d83db75df83384c6cd10087
Oct 29 22:35:37.115035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.114999    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:37.115132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.115067    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:37.115132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.115099    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:37.117034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.117007    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"]
Oct 29 22:35:37.181867 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:35:37.181839    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc12952_6dd5_4269_aa22_55b256dddbc1.slice/crio-22e46a4261420ae912f25c6b63e1ca057b17efc2a3742016dbc51df18686e8dd WatchSource:0}: Error finding container 22e46a4261420ae912f25c6b63e1ca057b17efc2a3742016dbc51df18686e8dd: Status 404 returned error can't find the container with id 22e46a4261420ae912f25c6b63e1ca057b17efc2a3742016dbc51df18686e8dd
Oct 29 22:35:37.185265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.185227    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:37.185363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.185313    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:37.185422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.185371    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:37.188084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.188063    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"]
Oct 29 22:35:37.774998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.774964    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" event={"ID":"ad3ea6e5-4dea-45ea-905e-b8091d29b209","Type":"ContainerStarted","Data":"419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec"}
Oct 29 22:35:37.775487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.775027    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" event={"ID":"ad3ea6e5-4dea-45ea-905e-b8091d29b209","Type":"ContainerStarted","Data":"dafc98cc33fd623604f07d603173c3e0cedc96962d83db75df83384c6cd10087"}
Oct 29 22:35:37.776929 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.776901    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" event={"ID":"9bc12952-6dd5-4269-aa22-55b256dddbc1","Type":"ContainerStarted","Data":"3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe"}
Oct 29 22:35:37.777030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.776939    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" event={"ID":"9bc12952-6dd5-4269-aa22-55b256dddbc1","Type":"ContainerStarted","Data":"22e46a4261420ae912f25c6b63e1ca057b17efc2a3742016dbc51df18686e8dd"}
Oct 29 22:35:37.789211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.789189    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:37.824552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.824509    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" podStartSLOduration=1.8244979749999999 podStartE2EDuration="1.824497975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:37.819213078 +0000 UTC m=+1279.843989233" watchObservedRunningTime="2026-10-29 22:35:37.824497975 +0000 UTC m=+1279.849274106"
Oct 29 22:35:37.843064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.843045    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:37.862945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:37.862893    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" podStartSLOduration=1.862877268 podStartE2EDuration="1.862877268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:37.861912019 +0000 UTC m=+1279.886688152" watchObservedRunningTime="2026-10-29 22:35:37.862877268 +0000 UTC m=+1279.887653401"
Oct 29 22:35:38.243698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:38.243671    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:38.791971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:38.791931    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:35:38.792615 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:38.792580    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" podUID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:35:38.844529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:38.844500    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:35:38.844728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:38.844711    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" podUID="9bc12952-6dd5-4269-aa22-55b256dddbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:35:39.663098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:39.663046    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"
Oct 29 22:35:39.663523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:39.663496    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"
Oct 29 22:35:39.819355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:39.814777    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:35:39.819355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:39.814838    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" podUID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:35:39.850861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:39.850559    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:35:39.852569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:39.852522    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" podUID="9bc12952-6dd5-4269-aa22-55b256dddbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:35:40.099349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:40.098740    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:40.790349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:40.789693    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:35:40.790349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:40.789786    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" podUID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:35:40.842895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:40.842856    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:35:40.843392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:40.842911    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" podUID="9bc12952-6dd5-4269-aa22-55b256dddbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:35:41.210141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.210111    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:41.638131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.638004    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"]
Oct 29 22:35:41.647195 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.647171    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.650481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.650456    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"]
Oct 29 22:35:41.717469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.692672    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1fa4b8b6-openshift-default-dockercfg-mnpx6\""
Oct 29 22:35:41.741353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.730370    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5af8478-08a0-4704-b55f-3692764b84e3-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.741667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.741635    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.741991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.741958    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.742124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.742108    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5af8478-08a0-4704-b55f-3692764b84e3-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.742236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.742223    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.742462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.742444    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twgb\" (UniqueName: \"kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-kube-api-access-9twgb\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.742623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.742609    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.742819 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.742801    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.742992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.742977    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.791349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.789475    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:35:41.791349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.789535    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" podUID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:35:41.791349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.791253    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.860033    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.860157    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5af8478-08a0-4704-b55f-3692764b84e3-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.860202    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.860421    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.860454    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5af8478-08a0-4704-b55f-3692764b84e3-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.860484    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.860582    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twgb\" (UniqueName: \"kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-kube-api-access-9twgb\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.860642    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.860733    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.861541    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.862041    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.862087    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" podUID="9bc12952-6dd5-4269-aa22-55b256dddbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:35:41.864352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.863145    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5af8478-08a0-4704-b55f-3692764b84e3-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.913525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.875666    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.913525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.876096    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.913525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.879702    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.913815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.913571    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.934595 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.915886    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5af8478-08a0-4704-b55f-3692764b84e3-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.953602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.953563    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:41.961368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.960297    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twgb\" (UniqueName: \"kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-kube-api-access-9twgb\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:41.976907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:41.976870    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:42.057278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.056773    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"]
Oct 29 22:35:42.061476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.061446    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2339b09e-7d4s9"]
Oct 29 22:35:42.156361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.154481    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="910fc8ce-5da1-4bb6-8cf5-cc120522aa59" path="/var/lib/kubelet/pods/910fc8ce-5da1-4bb6-8cf5-cc120522aa59/volumes"
Oct 29 22:35:42.265800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.265770    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:42.336372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.335299    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:42.724529 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:35:42.724483    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5af8478_08a0_4704_b55f_3692764b84e3.slice/crio-a228fa455a3baa8ed2f187f024ee2b071e864dc7f426adfece0ebcdd8387446b WatchSource:0}: Error finding container a228fa455a3baa8ed2f187f024ee2b071e864dc7f426adfece0ebcdd8387446b: Status 404 returned error can't find the container with id a228fa455a3baa8ed2f187f024ee2b071e864dc7f426adfece0ebcdd8387446b
Oct 29 22:35:42.727793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.727760    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:42.728057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.728033    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:42.728126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.728106    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:42.729795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.729768    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"]
Oct 29 22:35:42.810440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.809671    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:35:42.810440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.809726    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" podUID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:35:42.843354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.843307    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:35:42.843470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:42.843378    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" podUID="9bc12952-6dd5-4269-aa22-55b256dddbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:35:43.037215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.037116    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" event={"ID":"c5af8478-08a0-4704-b55f-3692764b84e3","Type":"ContainerStarted","Data":"5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2"}
Oct 29 22:35:43.037706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.037219    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" event={"ID":"c5af8478-08a0-4704-b55f-3692764b84e3","Type":"ContainerStarted","Data":"a228fa455a3baa8ed2f187f024ee2b071e864dc7f426adfece0ebcdd8387446b"}
Oct 29 22:35:43.128444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.127600    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" podStartSLOduration=2.127579339 podStartE2EDuration="2.127579339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:43.095778585 +0000 UTC m=+1285.120554753" watchObservedRunningTime="2026-10-29 22:35:43.127579339 +0000 UTC m=+1285.152355472"
Oct 29 22:35:43.274350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.267961    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:43.274350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.268075    4860 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:35:43.274350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.268117    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:35:43.633424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.633387    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:43.738730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.738701    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"
Oct 29 22:35:43.749308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.749273    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"]
Oct 29 22:35:43.756814 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.756718    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.762079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.762026    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"]
Oct 29 22:35:43.767469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.767418    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:43.769802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.769690    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.775401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.775380    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"]
Oct 29 22:35:43.789482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.785970    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-67ntj\""
Oct 29 22:35:43.795375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.794954    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:35:43.795375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.795006    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" podUID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:35:43.831017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.830932    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"]
Oct 29 22:35:43.860608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.850813    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:35:43.860608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.850857    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" podUID="9bc12952-6dd5-4269-aa22-55b256dddbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:35:43.862237 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.862205    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.863019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.862997    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.863215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.863197    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2c8v\" (UniqueName: \"kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-kube-api-access-d2c8v\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.863399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.863382    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.863551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.863534    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.864770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.864749    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0357f513-e857-424f-91f8-130b056ca135-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.864948 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.864930    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0357f513-e857-424f-91f8-130b056ca135-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.865152 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.865134    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.865309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.865293    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.865557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.865517    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.865758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.865738    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.865954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.865933    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.866116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.866099    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.866276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.866260    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.866647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.866451    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.866647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.866503    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.968500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.968048    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.968500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.968116    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwzrc\" (UniqueName: \"kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-kube-api-access-kwzrc\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.968500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.968158    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.968500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.968208    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.968500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.968266    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2c8v\" (UniqueName: \"kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-kube-api-access-d2c8v\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.968312    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.969256    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.969310    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0357f513-e857-424f-91f8-130b056ca135-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.969362    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0357f513-e857-424f-91f8-130b056ca135-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.969404    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.969433    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.969502    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.969577    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.969639    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.969683    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.969740    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.969790    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.969924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.969846    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.975122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.974412    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.975122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.974450    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0357f513-e857-424f-91f8-130b056ca135-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.975122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.974740    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.975388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.975224    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.975388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.975293    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.975926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.975536    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.975926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.975592    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.975926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.975826    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.976471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.976450    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.980651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.980612    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.981857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.981735    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.981857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.981807    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0357f513-e857-424f-91f8-130b056ca135-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.982208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.982177    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:43.997523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.997502    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.997868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.997842    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2c8v\" (UniqueName: \"kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-kube-api-access-d2c8v\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-82d8r\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:43.999390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:43.999367    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:44.004278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.004242    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:44.075138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.075082    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:44.075687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.075150    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwzrc\" (UniqueName: \"kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-kube-api-access-kwzrc\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:44.076065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.076033    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:44.096410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.096375    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwzrc\" (UniqueName: \"kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-kube-api-access-kwzrc\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cr7nq\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:44.159771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.159731    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:44.160781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.160753    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:44.276766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.275544    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:44.498086 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:35:44.498057    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0357f513_e857_424f_91f8_130b056ca135.slice/crio-a695d8e3af79d67f89ce2479e734388ab5fd746eac24c98a927f410e72eac768 WatchSource:0}: Error finding container a695d8e3af79d67f89ce2479e734388ab5fd746eac24c98a927f410e72eac768: Status 404 returned error can't find the container with id a695d8e3af79d67f89ce2479e734388ab5fd746eac24c98a927f410e72eac768
Oct 29 22:35:44.501156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.501127    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"]
Oct 29 22:35:44.501653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.501605    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:44.501775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.501706    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:44.501775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.501755    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:44.510472 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:35:44.510445    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac94c943_a3cf_4f8b_b653_eabafeff9c3d.slice/crio-bddb144e6f8e40fd44374fa4994bb2307f5dc6685d360b5444389ebdaa37eeeb WatchSource:0}: Error finding container bddb144e6f8e40fd44374fa4994bb2307f5dc6685d360b5444389ebdaa37eeeb: Status 404 returned error can't find the container with id bddb144e6f8e40fd44374fa4994bb2307f5dc6685d360b5444389ebdaa37eeeb
Oct 29 22:35:44.511711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.511693    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"]
Oct 29 22:35:44.521208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.521178    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:44.521300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.521274    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:44.521423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.521375    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:44.747265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.747167    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"
Oct 29 22:35:44.875348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.874642    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:35:44.875348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:44.874702    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" podUID="9bc12952-6dd5-4269-aa22-55b256dddbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:35:45.453356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.452564    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" event={"ID":"ac94c943-a3cf-4f8b-b653-eabafeff9c3d","Type":"ContainerStarted","Data":"6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea"}
Oct 29 22:35:45.453356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.452676    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" event={"ID":"ac94c943-a3cf-4f8b-b653-eabafeff9c3d","Type":"ContainerStarted","Data":"bddb144e6f8e40fd44374fa4994bb2307f5dc6685d360b5444389ebdaa37eeeb"}
Oct 29 22:35:45.495356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.486449    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" event={"ID":"0357f513-e857-424f-91f8-130b056ca135","Type":"ContainerStarted","Data":"e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30"}
Oct 29 22:35:45.495356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.486523    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" event={"ID":"0357f513-e857-424f-91f8-130b056ca135","Type":"ContainerStarted","Data":"a695d8e3af79d67f89ce2479e734388ab5fd746eac24c98a927f410e72eac768"}
Oct 29 22:35:45.495356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.486556    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:45.495356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.487500    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:45.502703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.502673    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:35:45.603347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.593167    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" podStartSLOduration=2.5931417420000002 podStartE2EDuration="2.593141742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:45.542705889 +0000 UTC m=+1287.567482022" watchObservedRunningTime="2026-10-29 22:35:45.593141742 +0000 UTC m=+1287.617917875"
Oct 29 22:35:45.649962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.649928    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:45.838137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.838012    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" podStartSLOduration=2.8379901480000003 podStartE2EDuration="2.837990148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:45.682991447 +0000 UTC m=+1287.707767605" watchObservedRunningTime="2026-10-29 22:35:45.837990148 +0000 UTC m=+1287.862766280"
Oct 29 22:35:45.842845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.842812    4860 patch_prober.go:29] interesting pod/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:35:45.843081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.843053    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" podUID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:35:45.952408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.942810    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:45.952408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.943837    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:45.960115 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:45.953784    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:35:46.007655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:46.007619    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:46.013451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:46.013370    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:46.037774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:46.037396    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:35:46.160558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:46.160470    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:46.162029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:46.161996    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:46.163236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:46.163205    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 29 22:35:46.163409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:46.163278    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 29 22:35:46.165740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:46.163402    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-82d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:35:46.165917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:46.165892    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" podUID="0357f513-e857-424f-91f8-130b056ca135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:35:47.160691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:47.160662    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-82d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:35:47.161150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:47.161124    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" podUID="0357f513-e857-424f-91f8-130b056ca135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:35:47.161482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:47.161459    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 29 22:35:47.161531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:47.161505    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 29 22:35:47.425446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:47.425236    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:47.619047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:47.618911    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:48.232991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:48.232958    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 29 22:35:48.233530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:48.233014    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 29 22:35:48.233530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:48.233358    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-82d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:35:48.233530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:48.233393    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" podUID="0357f513-e857-424f-91f8-130b056ca135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:35:48.238200 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:35:48.238076    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:35:49.160723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:49.160691    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-82d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:35:49.160903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:49.160756    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" podUID="0357f513-e857-424f-91f8-130b056ca135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:35:49.161024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:49.161001    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 29 22:35:49.161074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:49.161038    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 29 22:35:49.215351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:49.215293    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:50.160047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:50.160015    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-82d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:35:50.160460 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:50.160064    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" podUID="0357f513-e857-424f-91f8-130b056ca135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:35:50.161293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:50.161271    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 29 22:35:50.161377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:50.161313    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 29 22:35:50.726465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:50.726404    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:50.842716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:50.842687    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:50.964438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:50.964406    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:51.160826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:51.160742    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-82d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:35:51.160826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:51.160806    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" podUID="0357f513-e857-424f-91f8-130b056ca135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:35:51.161248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:51.161105    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 29 22:35:51.161248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:51.161152    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 29 22:35:52.168361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.168274    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-82d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:35:52.169762 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.168379    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" podUID="0357f513-e857-424f-91f8-130b056ca135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:35:52.169762 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.168274    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 29 22:35:52.169762 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.168463    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 29 22:35:52.253210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.253173    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"]
Oct 29 22:35:52.261688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.261660    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.278931 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.278844    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"]
Oct 29 22:35:52.279223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.279201    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 29 22:35:52.279371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.279234    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rgwvk\""
Oct 29 22:35:52.279467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.279281    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 29 22:35:52.279547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.279317    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 29 22:35:52.286595 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.286570    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8bs\" (UniqueName: \"kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-kube-api-access-kn8bs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.286864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.286846    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5567035-eebe-4ab7-8875-876f89630445-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.287062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.287046    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.287243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.287216    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.287450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.287425    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.287540 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.287468    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.287606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.287584    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.390427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.390385    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.390816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.390785    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.395702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.395674    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.399590 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.392393    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.399910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.399883    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8bs\" (UniqueName: \"kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-kube-api-access-kn8bs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.400655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.400632    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5567035-eebe-4ab7-8875-876f89630445-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.400760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.400687    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.400760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.400717    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.406448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.406417    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.407876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.407854    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5567035-eebe-4ab7-8875-876f89630445-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.408917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.408897    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.413223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.411259    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.413223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.412863    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.423974 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.423906    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8bs\" (UniqueName: \"kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-kube-api-access-kn8bs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z794f\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.547027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.546980    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:52.589942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.589910    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:52.953385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.953342    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"]
Oct 29 22:35:52.960784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:52.960732    4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:35:53.161065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:53.161020    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 29 22:35:53.161243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:53.161068    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 29 22:35:53.674678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:53.674638    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f" event={"ID":"d5567035-eebe-4ab7-8875-876f89630445","Type":"ContainerStarted","Data":"384c034e4bb907e48b6339edcf4c102120952aa3efc955c756619a347ca0b9b5"}
Oct 29 22:35:54.042348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:54.029716    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:54.042348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:54.031213    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:54.042348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:54.040297    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:35:54.172779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:54.172697    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:54.476526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:54.476494    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:54.477687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:54.477661    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:54.478719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:54.478694    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 22:35:55.945427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:55.945378    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:56.035573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:56.035535    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"]
Oct 29 22:35:56.070484 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:56.070437    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e6b7cf-9k9gq"]
Oct 29 22:35:56.118233 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:56.118196    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2018efa-a97e-4104-92ff-b97be01361e4" path="/var/lib/kubelet/pods/e2018efa-a97e-4104-92ff-b97be01361e4/volumes"
Oct 29 22:35:56.221015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:56.220977    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:56.221133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:56.221044    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:35:56.728872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:56.728842    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f" event={"ID":"d5567035-eebe-4ab7-8875-876f89630445","Type":"ContainerStarted","Data":"c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82"}
Oct 29 22:35:56.729055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:56.728937    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:35:56.775126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:56.775075    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f" podStartSLOduration=1.516644122 podStartE2EDuration="4.775060728s" totalImagesPullingTime="3.258416606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:56.773518044 +0000 UTC m=+1298.798294186" watchObservedRunningTime="2026-10-29 22:35:56.775060728 +0000 UTC m=+1298.799836860"
Oct 29 22:35:57.497363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:57.494617    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:57.740851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:57.740807    4860 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-z794f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:35:57.741038 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:57.740874    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f" podUID="d5567035-eebe-4ab7-8875-876f89630445" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:35:58.717220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:58.717182    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:59.032046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:59.031968    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"]
Oct 29 22:35:59.045355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:59.045314    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b45a2214-j9pjp"]
Oct 29 22:35:59.155383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:59.155351    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:59.585265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:59.585232    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:35:59.701429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:35:59.701360    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:00.115566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:00.115537    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83847c71-2143-44d7-b57e-1dfab1cfad9f" path="/var/lib/kubelet/pods/83847c71-2143-44d7-b57e-1dfab1cfad9f/volumes"
Oct 29 22:36:00.190376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:00.190345    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:00.740294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:00.740262    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 22:36:00.802768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:00.802736    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:02.616297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:02.616265    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:04.315893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:04.315854    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:05.995234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:05.995201    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:06.037987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:06.037948    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"]
Oct 29 22:36:06.053566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:06.053534    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7898f8-hgfmb"]
Oct 29 22:36:06.115442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:06.115405    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb79b8a-b606-4347-9015-af441b1992a0" path="/var/lib/kubelet/pods/8eb79b8a-b606-4347-9015-af441b1992a0/volumes"
Oct 29 22:36:06.569473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:06.569437    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:07.712795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:07.712758    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:07.834929 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:07.834893    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:08.332931 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:08.332887    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:09.823764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:09.823713    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:10.746124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:10.746072    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:11.521011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:11.520955    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:11.646365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:11.646284    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:12.058182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:12.058111    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"]
Oct 29 22:36:12.064433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:12.064341    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ae3e8a-wp6mn"]
Oct 29 22:36:12.113231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:12.113198    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:12.125725 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:12.124610    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8" path="/var/lib/kubelet/pods/68fc3b71-f6ac-4940-bd0d-ef4b99e9eaf8/volumes"
Oct 29 22:36:12.243979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:12.243931    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:12.397409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:12.391852    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:13.290189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:13.290125    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:13.427928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:13.427847    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:13.579434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:13.574175    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:14.995901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:14.995853    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:15.015598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.015382    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"]
Oct 29 22:36:15.036994 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.023803    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:15.036994 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.033513    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"]
Oct 29 22:36:15.043996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.043964    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:36:15.044167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.044124    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:36:15.044241 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.044212    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:36:15.044579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.044488    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:36:15.117542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.117503    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625caf72-c0df-499f-b684-ff559d204460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9684486f-xpv5k\" (UID: \"625caf72-c0df-499f-b684-ff559d204460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:15.117807 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.117789    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9qn7\" (UniqueName: \"kubernetes.io/projected/625caf72-c0df-499f-b684-ff559d204460-kube-api-access-z9qn7\") pod \"maas-api-revoke-sub-9684486f-xpv5k\" (UID: \"625caf72-c0df-499f-b684-ff559d204460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:15.117947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.117935    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625caf72-c0df-499f-b684-ff559d204460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9684486f-xpv5k\" (UID: \"625caf72-c0df-499f-b684-ff559d204460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:15.219517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.219484    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625caf72-c0df-499f-b684-ff559d204460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9684486f-xpv5k\" (UID: \"625caf72-c0df-499f-b684-ff559d204460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:15.219741 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.219726    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625caf72-c0df-499f-b684-ff559d204460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9684486f-xpv5k\" (UID: \"625caf72-c0df-499f-b684-ff559d204460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:15.219820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.219809    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9qn7\" (UniqueName: \"kubernetes.io/projected/625caf72-c0df-499f-b684-ff559d204460-kube-api-access-z9qn7\") pod \"maas-api-revoke-sub-9684486f-xpv5k\" (UID: \"625caf72-c0df-499f-b684-ff559d204460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:15.220639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.220608    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625caf72-c0df-499f-b684-ff559d204460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9684486f-xpv5k\" (UID: \"625caf72-c0df-499f-b684-ff559d204460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:15.221950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.221929    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625caf72-c0df-499f-b684-ff559d204460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9684486f-xpv5k\" (UID: \"625caf72-c0df-499f-b684-ff559d204460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:15.241049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.241021    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9qn7\" (UniqueName: \"kubernetes.io/projected/625caf72-c0df-499f-b684-ff559d204460-kube-api-access-z9qn7\") pod \"maas-api-revoke-sub-9684486f-xpv5k\" (UID: \"625caf72-c0df-499f-b684-ff559d204460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:15.371425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.367500    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:15.699602 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:36:15.699564    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625caf72_c0df_499f_b684_ff559d204460.slice/crio-b68cb310f6bcf4f44067ff4995fb400c299a3b98a99ebd060ac8f6e1ad2b9234 WatchSource:0}: Error finding container b68cb310f6bcf4f44067ff4995fb400c299a3b98a99ebd060ac8f6e1ad2b9234: Status 404 returned error can't find the container with id b68cb310f6bcf4f44067ff4995fb400c299a3b98a99ebd060ac8f6e1ad2b9234
Oct 29 22:36:15.701459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.701437    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"]
Oct 29 22:36:15.994214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:15.994126    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k" event={"ID":"625caf72-c0df-499f-b684-ff559d204460","Type":"ContainerStarted","Data":"b68cb310f6bcf4f44067ff4995fb400c299a3b98a99ebd060ac8f6e1ad2b9234"}
Oct 29 22:36:16.640090 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:16.639884    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:17.006594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:17.006561    4860 generic.go:356] "Generic (PLEG): container finished" podID="625caf72-c0df-499f-b684-ff559d204460" containerID="403af61a7f99e774d815ec752efba0a9265f12701194526a652e19195a7c63fe" exitCode=0
Oct 29 22:36:17.006734 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:17.006592    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k" event={"ID":"625caf72-c0df-499f-b684-ff559d204460","Type":"ContainerDied","Data":"403af61a7f99e774d815ec752efba0a9265f12701194526a652e19195a7c63fe"}
Oct 29 22:36:18.067067 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:36:18.067030    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:36:18.209483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:18.209456    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:18.277467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:18.277381    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625caf72-c0df-499f-b684-ff559d204460-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625caf72-c0df-499f-b684-ff559d204460-maas-api-xks-ca\") pod \"625caf72-c0df-499f-b684-ff559d204460\" (UID: \"625caf72-c0df-499f-b684-ff559d204460\") "
Oct 29 22:36:18.277715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:18.277692    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625caf72-c0df-499f-b684-ff559d204460-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625caf72-c0df-499f-b684-ff559d204460-maas-api-openshift-ca\") pod \"625caf72-c0df-499f-b684-ff559d204460\" (UID: \"625caf72-c0df-499f-b684-ff559d204460\") "
Oct 29 22:36:18.278853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:18.278801    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625caf72-c0df-499f-b684-ff559d204460-kube-api-access-z9qn7\" (UniqueName: \"kubernetes.io/projected/625caf72-c0df-499f-b684-ff559d204460-kube-api-access-z9qn7\") pod \"625caf72-c0df-499f-b684-ff559d204460\" (UID: \"625caf72-c0df-499f-b684-ff559d204460\") "
Oct 29 22:36:18.281761 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:18.281726    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625caf72-c0df-499f-b684-ff559d204460-maas-api-xks-ca" pod "625caf72-c0df-499f-b684-ff559d204460" (UID: "625caf72-c0df-499f-b684-ff559d204460"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:36:18.281885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:18.281866    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625caf72-c0df-499f-b684-ff559d204460-kube-api-access-z9qn7" pod "625caf72-c0df-499f-b684-ff559d204460" (UID: "625caf72-c0df-499f-b684-ff559d204460"). InnerVolumeSpecName "kube-api-access-z9qn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:36:18.283853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:18.283762    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625caf72-c0df-499f-b684-ff559d204460-maas-api-openshift-ca" pod "625caf72-c0df-499f-b684-ff559d204460" (UID: "625caf72-c0df-499f-b684-ff559d204460"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:36:18.348345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:18.348294    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:18.381533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:18.381486    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625caf72-c0df-499f-b684-ff559d204460-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:36:18.381533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:18.381528    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9qn7\" (UniqueName: \"kubernetes.io/projected/625caf72-c0df-499f-b684-ff559d204460-kube-api-access-z9qn7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:36:18.381533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:18.381541    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625caf72-c0df-499f-b684-ff559d204460-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:36:18.688503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:18.688471    4860 scope.go:122] "RemoveContainer" containerID="466869ed1c5639b12716135145eda11d00b0d3eedbafc9c773f881897aef136e"
Oct 29 22:36:19.033062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:19.032964    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"
Oct 29 22:36:19.033062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:19.032993    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k" event={"ID":"625caf72-c0df-499f-b684-ff559d204460","Type":"ContainerDied","Data":"b68cb310f6bcf4f44067ff4995fb400c299a3b98a99ebd060ac8f6e1ad2b9234"}
Oct 29 22:36:19.033062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:19.033042    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b68cb310f6bcf4f44067ff4995fb400c299a3b98a99ebd060ac8f6e1ad2b9234"
Oct 29 22:36:19.849012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:19.848977    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:19.974380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:19.973190    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:20.038844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:20.038795    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"]
Oct 29 22:36:20.051127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:20.051090    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-834c33af-65b7f"]
Oct 29 22:36:20.129831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:20.129754    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936d0568-d56a-4fd0-99c5-51f769ffd518" path="/var/lib/kubelet/pods/936d0568-d56a-4fd0-99c5-51f769ffd518/volumes"
Oct 29 22:36:21.374574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:21.374544    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:22.066746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:22.066670    4860 generic.go:356] "Generic (PLEG): container finished" podID="a551643a-4108-4d0d-be08-6add34edffb1" containerID="3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386" exitCode=0
Oct 29 22:36:22.066895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:22.066740    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" event={"ID":"a551643a-4108-4d0d-be08-6add34edffb1","Type":"ContainerDied","Data":"3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386"}
Oct 29 22:36:22.067123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:22.067109    4860 scope.go:122] "RemoveContainer" containerID="3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386"
Oct 29 22:36:22.068672 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:22.068647    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:36:22.068749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:22.068714    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:36:22.068749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:22.068744    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:36:23.119447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:23.119410    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" event={"ID":"a551643a-4108-4d0d-be08-6add34edffb1","Type":"ContainerStarted","Data":"56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3"}
Oct 29 22:36:23.141634 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:23.141602    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:23.596420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:23.596383    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:36:23.599573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:23.599538    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:36:23.599739 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:23.599595    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:36:24.590350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:24.589752    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:36:24.590350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:24.589811    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:36:24.605895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:24.605869    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:36:24.652094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:24.652061    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:25.035985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.035953    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"]
Oct 29 22:36:25.049683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.049653    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e7dbcce-j65pg"]
Oct 29 22:36:25.309355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.309244    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:25.591729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.591640    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:36:25.592281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.592252    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:36:25.802389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.802356    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"]
Oct 29 22:36:25.803654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.803624    4860 cpu_manager.go:414] "containerMap: removing container" podUID="625caf72-c0df-499f-b684-ff559d204460" containerName="revoke-keys"
Oct 29 22:36:25.803654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.803650    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625caf72-c0df-499f-b684-ff559d204460" containerName="revoke-keys"
Oct 29 22:36:25.803870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.803860    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="625caf72-c0df-499f-b684-ff559d204460" containerName="revoke-keys"
Oct 29 22:36:25.807486 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.807460    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.809292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.809265    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"]
Oct 29 22:36:25.824820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.824797    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-3bd0f6-openshift-default-dockercfg-cmm6g\""
Oct 29 22:36:25.838021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.837995    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j99br\" (UniqueName: \"kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-kube-api-access-j99br\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.838117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.838048    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.838159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.838119    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.838159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.838150    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.838313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.838289    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.838423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.838374    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.838557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.838535    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.838624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.838605    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.838664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.838650    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e56389b-3fc6-4877-9034-efdf79f2df59-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.940608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.940574    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.940770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.940656    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.940770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.940700    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e56389b-3fc6-4877-9034-efdf79f2df59-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.940898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.940876    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j99br\" (UniqueName: \"kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-kube-api-access-j99br\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.941027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.941010    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.941027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.941015    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-data\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.941189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.941053    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.941189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.941084    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.941189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.941181    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.941610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.941213    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.941610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.941315    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-credential-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.941610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.941452    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e56389b-3fc6-4877-9034-efdf79f2df59-istiod-ca-cert\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.941777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.941608    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-socket\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.941777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.941746    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-certs\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.943929 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.943904    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-envoy\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.944113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.944096    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-podinfo\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.962024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.961997    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-token\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:25.962155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:25.962132    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j99br\" (UniqueName: \"kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-kube-api-access-j99br\") pod \"e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") " pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:26.030977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:26.030942    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"]
Oct 29 22:36:26.036765 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:26.036740    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1234ae-lvjls"]
Oct 29 22:36:26.114008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:26.113978    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0935a1-4bdb-4990-bd2c-3145cff30d9c" path="/var/lib/kubelet/pods/2a0935a1-4bdb-4990-bd2c-3145cff30d9c/volumes"
Oct 29 22:36:26.114567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:26.114552    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8243fc-764b-46d3-98f4-65944b4bb80b" path="/var/lib/kubelet/pods/cc8243fc-764b-46d3-98f4-65944b4bb80b/volumes"
Oct 29 22:36:26.126020 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:26.125991    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:26.297219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:26.297179    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:26.501922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:26.501489    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:36:26.501922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:26.501565    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:36:26.501922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:26.501593    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:36:26.503999 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:26.503968    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"]
Oct 29 22:36:26.597847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:26.597812    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:36:26.598375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:26.597871    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:36:27.161390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:27.161353    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k" event={"ID":"9e56389b-3fc6-4877-9034-efdf79f2df59","Type":"ContainerStarted","Data":"32f9a080bb6d1a07d14a80c3a751490486cfb7bf71149e1a830fc9ba7ce5f41f"}
Oct 29 22:36:27.161592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:27.161415    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k" event={"ID":"9e56389b-3fc6-4877-9034-efdf79f2df59","Type":"ContainerStarted","Data":"ea5e293a09bff846c206e9167e258062cd5255cba58e2e0ddad5ef5e1bcb8e59"}
Oct 29 22:36:27.249437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:27.249370    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k" podStartSLOduration=2.249350535 podStartE2EDuration="2.249350535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:27.231031391 +0000 UTC m=+1329.255807575" watchObservedRunningTime="2026-10-29 22:36:27.249350535 +0000 UTC m=+1329.274126670"
Oct 29 22:36:27.604360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:27.603457    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:36:27.604360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:27.603532    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:36:28.146201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:28.145582    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 29 22:36:28.146201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:28.145644    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k" podUID="9e56389b-3fc6-4877-9034-efdf79f2df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 29 22:36:28.153225 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:28.152495    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:28.237376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:28.234286    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:28.593997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:28.593948    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:36:28.594208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:28.594010    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:36:29.127088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:29.127038    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 29 22:36:29.127477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:29.127105    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k" podUID="9e56389b-3fc6-4877-9034-efdf79f2df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 29 22:36:29.597988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:29.597958    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:36:29.599306 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:29.599282    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:36:29.861420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:29.861342    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:30.127246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:30.127173    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 29 22:36:30.127246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:30.127222    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k" podUID="9e56389b-3fc6-4877-9034-efdf79f2df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 29 22:36:31.127286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:31.127253    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 29 22:36:31.127696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:31.127301    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k" podUID="9e56389b-3fc6-4877-9034-efdf79f2df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 29 22:36:31.408446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:31.408370    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:31.520371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:31.520307    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:32.126501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:32.126470    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 29 22:36:32.126679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:32.126518    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k" podUID="9e56389b-3fc6-4877-9034-efdf79f2df59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 29 22:36:32.679190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:32.679151    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:33.123648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:33.123606    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:33.132075 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:33.132039    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:33.132956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:33.132932    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:33.133573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:33.133550    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 22:36:33.263671 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:33.263632    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:33.455778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:33.455734    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:35.029289 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:35.029260    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"]
Oct 29 22:36:35.043423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:35.043385    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79db70ec-9mdvf"]
Oct 29 22:36:35.268801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:35.268755    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:36.033233 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:36.033199    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"]
Oct 29 22:36:36.046502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:36.046472    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4374a0-lkg4p"]
Oct 29 22:36:36.114313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:36.114272    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d78a9eb3-aeb2-40d5-8a5c-31be12e6b380" path="/var/lib/kubelet/pods/d78a9eb3-aeb2-40d5-8a5c-31be12e6b380/volumes"
Oct 29 22:36:36.114908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:36.114884    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2068b1-9c86-4569-b49d-606acafa57ee" path="/var/lib/kubelet/pods/dd2068b1-9c86-4569-b49d-606acafa57ee/volumes"
Oct 29 22:36:36.388781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:36.388687    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:36.936602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:36.936569    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:37.969276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:37.969242    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:38.040585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:38.040546    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"]
Oct 29 22:36:38.051672 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:38.051643    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd631cb-6sp22"]
Oct 29 22:36:38.107080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:38.107032    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:38.127847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:38.127812    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924a9308-baf9-4f32-aa16-ba545a327ee5" path="/var/lib/kubelet/pods/924a9308-baf9-4f32-aa16-ba545a327ee5/volumes"
Oct 29 22:36:38.287648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:38.287537    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:39.142310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:39.142271    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:39.514745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:39.514706    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:39.987476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:39.987424    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:40.291424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:40.291304    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:40.424948 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:40.424913    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:40.883942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:40.883880    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:41.025719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:41.025682    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:42.535288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:42.535253    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:44.337871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:44.337827    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:45.323049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:45.323007    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:45.452820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:45.452757    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:46.043542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.043504    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:46.185457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.185420    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:46.429347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.429197    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"]
Oct 29 22:36:46.435942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.435915    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.461635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.461605    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ee886f-openshift-default-dockercfg-rktsp\""
Oct 29 22:36:46.481618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.481587    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"]
Oct 29 22:36:46.535259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.535220    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-data\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.535420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.535385    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-istio-token\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.535545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.535526    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-credential-socket\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.535666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.535644    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551bb653-c966-43cc-8f02-007606849b1f-istio-podinfo\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.535788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.535767    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4db85\" (UniqueName: \"kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-kube-api-access-4db85\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.535827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.535808    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-certs\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.535989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.535972    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551bb653-c966-43cc-8f02-007606849b1f-istiod-ca-cert\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.536030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.536010    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-envoy\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.536107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.536092    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-socket\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.566525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.566496    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:46.637200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.637166    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551bb653-c966-43cc-8f02-007606849b1f-istio-podinfo\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.637414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.637238    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4db85\" (UniqueName: \"kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-kube-api-access-4db85\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.637414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.637257    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-certs\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.637414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.637364    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551bb653-c966-43cc-8f02-007606849b1f-istiod-ca-cert\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.637414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.637399    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-envoy\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.637660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.637447    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-socket\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.637660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.637524    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-data\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.637769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.637674    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-istio-token\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.637827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.637779    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-credential-socket\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.637827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.637785    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-certs\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.637933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.637877    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-socket\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.638021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.638001    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-data\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.638198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.638176    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551bb653-c966-43cc-8f02-007606849b1f-istiod-ca-cert\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.638291 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.638178    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-credential-socket\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.640042 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.640014    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551bb653-c966-43cc-8f02-007606849b1f-istio-podinfo\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.640042 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.640038    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-envoy\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.659642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.659618    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-istio-token\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.659742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.659701    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4db85\" (UniqueName: \"kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-kube-api-access-4db85\") pod \"e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") " pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:46.726752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.726650    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:46.805359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:46.804630    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:47.140081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:47.137249    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:47.243502 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:36:47.243473    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551bb653_c966_43cc_8f02_007606849b1f.slice/crio-d00312cd0e4a4aa7ba135dcfc217eaedd874903a68840195b0187fe1accf0712 WatchSource:0}: Error finding container d00312cd0e4a4aa7ba135dcfc217eaedd874903a68840195b0187fe1accf0712: Status 404 returned error can't find the container with id d00312cd0e4a4aa7ba135dcfc217eaedd874903a68840195b0187fe1accf0712
Oct 29 22:36:47.247000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:47.246962    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:36:47.247117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:47.247051    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:36:47.247117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:47.247097    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:36:47.249560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:47.249535    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"]
Oct 29 22:36:47.274384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:47.274344    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:47.622125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:47.622022    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" event={"ID":"551bb653-c966-43cc-8f02-007606849b1f","Type":"ContainerStarted","Data":"0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813"}
Oct 29 22:36:47.625469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:47.625440    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" event={"ID":"551bb653-c966-43cc-8f02-007606849b1f","Type":"ContainerStarted","Data":"d00312cd0e4a4aa7ba135dcfc217eaedd874903a68840195b0187fe1accf0712"}
Oct 29 22:36:47.722236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:47.722187    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" podStartSLOduration=1.722172457 podStartE2EDuration="1.722172457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:47.703280017 +0000 UTC m=+1349.728056150" watchObservedRunningTime="2026-10-29 22:36:47.722172457 +0000 UTC m=+1349.746948587"
Oct 29 22:36:47.811346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:47.807459    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:48.245401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.243534    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"]
Oct 29 22:36:48.245401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.243587    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-813dc797-hsvkp"]
Oct 29 22:36:48.245401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.243613    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"]
Oct 29 22:36:48.252371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.249846    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"]
Oct 29 22:36:48.252371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.249972    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:48.289360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.281096    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:36:48.311394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.311133    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:36:48.311394 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:36:48.311311    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:36:48.311639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.311465    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:36:48.311639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.311558    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:36:48.353137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.353105    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:48.421350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.419642    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99ca7ac1-xvdb9\" (UID: \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:48.421350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.419783    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99ca7ac1-xvdb9\" (UID: \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:48.421350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.420003    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxzsz\" (UniqueName: \"kubernetes.io/projected/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-kube-api-access-hxzsz\") pod \"maas-api-revoke-sub-99ca7ac1-xvdb9\" (UID: \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:48.523916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.523261    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99ca7ac1-xvdb9\" (UID: \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:48.523916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.523586    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxzsz\" (UniqueName: \"kubernetes.io/projected/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-kube-api-access-hxzsz\") pod \"maas-api-revoke-sub-99ca7ac1-xvdb9\" (UID: \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:48.523916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.523835    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99ca7ac1-xvdb9\" (UID: \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:48.530375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.524753    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99ca7ac1-xvdb9\" (UID: \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:48.549388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.547215    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99ca7ac1-xvdb9\" (UID: \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:48.595667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.581272    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxzsz\" (UniqueName: \"kubernetes.io/projected/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-kube-api-access-hxzsz\") pod \"maas-api-revoke-sub-99ca7ac1-xvdb9\" (UID: \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:48.628357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.623926    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:48.838400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:48.833526    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:49.858413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:49.858384    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:49.859203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:49.859186    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:36:49.945683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:49.943540    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"]
Oct 29 22:36:50.310348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:50.308341    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07177684-d8bb-49e3-9634-77c995f3840f" path="/var/lib/kubelet/pods/07177684-d8bb-49e3-9634-77c995f3840f/volumes"
Oct 29 22:36:50.322346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:50.321818    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:50.489107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:50.489074    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:50.896381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:50.894093    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9" event={"ID":"de0281ae-c434-4ebd-b8b3-bb6cf61693c3","Type":"ContainerStarted","Data":"2091176f66ba292007c275931fb1ab693fc974f360145f470309d511255136df"}
Oct 29 22:36:51.040648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:51.040613    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"]
Oct 29 22:36:51.046944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:51.046908    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b97c28ca-h9tz6"]
Oct 29 22:36:51.905882 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:51.905799    4860 generic.go:356] "Generic (PLEG): container finished" podID="de0281ae-c434-4ebd-b8b3-bb6cf61693c3" containerID="dd8eb39284069a1d964619c2021379a401fe5a5a245ac49d963be2da27c1ab21" exitCode=0
Oct 29 22:36:51.906380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:51.905883    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9" event={"ID":"de0281ae-c434-4ebd-b8b3-bb6cf61693c3","Type":"ContainerDied","Data":"dd8eb39284069a1d964619c2021379a401fe5a5a245ac49d963be2da27c1ab21"}
Oct 29 22:36:52.046722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.046672    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:52.114433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.114398    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec5d62ba-ce99-47ca-83c3-82bacd5b4d63" path="/var/lib/kubelet/pods/ec5d62ba-ce99-47ca-83c3-82bacd5b4d63/volumes"
Oct 29 22:36:52.238382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.238346    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"]
Oct 29 22:36:52.243569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.243546    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"
Oct 29 22:36:52.253154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.253131    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:52.261822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.261803    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ee886f\""
Oct 29 22:36:52.262354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.262339    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ee886f-dockercfg-dcs78\""
Oct 29 22:36:52.306820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.306026    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"]
Oct 29 22:36:52.444352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.441290    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61010f1-3be5-4385-bf1d-3cb8331b6058-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469\" (UID: \"a61010f1-3be5-4385-bf1d-3cb8331b6058\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"
Oct 29 22:36:52.444352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.441504    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxxr\" (UniqueName: \"kubernetes.io/projected/a61010f1-3be5-4385-bf1d-3cb8331b6058-kube-api-access-qnxxr\") pod \"payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469\" (UID: \"a61010f1-3be5-4385-bf1d-3cb8331b6058\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"
Oct 29 22:36:52.455411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.454073    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:52.560798 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.560697    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61010f1-3be5-4385-bf1d-3cb8331b6058-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469\" (UID: \"a61010f1-3be5-4385-bf1d-3cb8331b6058\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"
Oct 29 22:36:52.619891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.561928    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61010f1-3be5-4385-bf1d-3cb8331b6058-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469\" (UID: \"a61010f1-3be5-4385-bf1d-3cb8331b6058\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"
Oct 29 22:36:52.620154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.619990    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxxr\" (UniqueName: \"kubernetes.io/projected/a61010f1-3be5-4385-bf1d-3cb8331b6058-kube-api-access-qnxxr\") pod \"payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469\" (UID: \"a61010f1-3be5-4385-bf1d-3cb8331b6058\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"
Oct 29 22:36:52.645635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.645594    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxxr\" (UniqueName: \"kubernetes.io/projected/a61010f1-3be5-4385-bf1d-3cb8331b6058-kube-api-access-qnxxr\") pod \"payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469\" (UID: \"a61010f1-3be5-4385-bf1d-3cb8331b6058\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"
Oct 29 22:36:52.859410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:52.859309    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"
Oct 29 22:36:53.248567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.248540    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:53.298429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.298404    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"]
Oct 29 22:36:53.340061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.340035    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-xks-ca\") pod \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\" (UID: \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\") "
Oct 29 22:36:53.340196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.340095    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-kube-api-access-hxzsz\" (UniqueName: \"kubernetes.io/projected/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-kube-api-access-hxzsz\") pod \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\" (UID: \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\") "
Oct 29 22:36:53.340283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.340266    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-openshift-ca\") pod \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\" (UID: \"de0281ae-c434-4ebd-b8b3-bb6cf61693c3\") "
Oct 29 22:36:53.340947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.340877    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-openshift-ca" pod "de0281ae-c434-4ebd-b8b3-bb6cf61693c3" (UID: "de0281ae-c434-4ebd-b8b3-bb6cf61693c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:36:53.343629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.343577    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-kube-api-access-hxzsz" pod "de0281ae-c434-4ebd-b8b3-bb6cf61693c3" (UID: "de0281ae-c434-4ebd-b8b3-bb6cf61693c3"). InnerVolumeSpecName "kube-api-access-hxzsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:36:53.350732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.350709    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-xks-ca" pod "de0281ae-c434-4ebd-b8b3-bb6cf61693c3" (UID: "de0281ae-c434-4ebd-b8b3-bb6cf61693c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:36:53.442944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.442907    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:36:53.442944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.442943    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxzsz\" (UniqueName: \"kubernetes.io/projected/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-kube-api-access-hxzsz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:36:53.443113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.442955    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0281ae-c434-4ebd-b8b3-bb6cf61693c3-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:36:53.957390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.953493    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"
Oct 29 22:36:53.957390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.953718    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9" event={"ID":"de0281ae-c434-4ebd-b8b3-bb6cf61693c3","Type":"ContainerDied","Data":"2091176f66ba292007c275931fb1ab693fc974f360145f470309d511255136df"}
Oct 29 22:36:53.957390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.953765    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2091176f66ba292007c275931fb1ab693fc974f360145f470309d511255136df"
Oct 29 22:36:53.972033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.971997    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469" event={"ID":"a61010f1-3be5-4385-bf1d-3cb8331b6058","Type":"ContainerStarted","Data":"9eb112ab7c1a39e891421c9025b1fba29b20fee55e2b8a70902bcb4855c2858c"}
Oct 29 22:36:53.972169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.972056    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469" event={"ID":"a61010f1-3be5-4385-bf1d-3cb8331b6058","Type":"ContainerStarted","Data":"6cdc8ee215f87f1eb0b060f89e1f09f73573ad930c60c1111ed7019e2bbd4906"}
Oct 29 22:36:53.972307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:53.972290    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"
Oct 29 22:36:54.037893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:54.037840    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469" podStartSLOduration=2.037825719 podStartE2EDuration="2.037825719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:54.030043958 +0000 UTC m=+1356.054820088" watchObservedRunningTime="2026-10-29 22:36:54.037825719 +0000 UTC m=+1356.062601849"
Oct 29 22:36:54.174095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:54.174066    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:55.810569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:55.810539    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:57.452440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:57.452378    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:58.016449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:58.016421    4860 generic.go:356] "Generic (PLEG): container finished" podID="c5af8478-08a0-4704-b55f-3692764b84e3" containerID="5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2" exitCode=0
Oct 29 22:36:58.016621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:58.016501    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" event={"ID":"c5af8478-08a0-4704-b55f-3692764b84e3","Type":"ContainerDied","Data":"5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2"}
Oct 29 22:36:58.016881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:58.016866    4860 scope.go:122] "RemoveContainer" containerID="5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2"
Oct 29 22:36:58.018630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:58.018599    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:36:58.018774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:58.018684    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:36:58.018774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:58.018733    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:36:58.413663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:58.413588    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:58.995655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:58.993912    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:36:59.044833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:59.044743    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" event={"ID":"c5af8478-08a0-4704-b55f-3692764b84e3","Type":"ContainerStarted","Data":"720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35"}
Oct 29 22:36:59.266244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:59.266165    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:36:59.292565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:59.292500    4860 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:36:59.292769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:36:59.292577    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:37:00.267030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:00.266999    4860 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:37:00.267560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:00.267063    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:37:00.466244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:00.466211    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:37:00.751760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:00.751729    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:01.031150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:01.031069    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"]
Oct 29 22:37:01.044310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:01.044268    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb2ac814-nfhp4"]
Oct 29 22:37:01.266472 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:01.266437    4860 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:37:01.266646 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:01.266489    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:37:02.032570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:02.032528    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"]
Oct 29 22:37:02.046384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:02.046355    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a458a371-xw9ch"]
Oct 29 22:37:02.114176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:02.114142    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ffa1a1-4cb6-490a-a913-ad7c27348866" path="/var/lib/kubelet/pods/14ffa1a1-4cb6-490a-a913-ad7c27348866/volumes"
Oct 29 22:37:02.114768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:02.114751    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2230d571-c982-421c-8515-fac6021c718f" path="/var/lib/kubelet/pods/2230d571-c982-421c-8515-fac6021c718f/volumes"
Oct 29 22:37:02.293511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:02.293419    4860 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:37:02.293845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:02.293774    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:37:02.304243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:02.304217    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:02.435716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:02.435677    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:03.033195 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:03.033164    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"]
Oct 29 22:37:03.044548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:03.044520    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb742ee-tzl2m"]
Oct 29 22:37:03.266240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:03.266208    4860 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:37:03.266441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:03.266263    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:37:03.983303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:03.983262    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:04.120167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:04.120128    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b703e63f-9cee-4556-be46-450232faaa43" path="/var/lib/kubelet/pods/b703e63f-9cee-4556-be46-450232faaa43/volumes"
Oct 29 22:37:04.227973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:04.227941    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:04.386884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:04.386797    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:37:04.388282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:04.388255    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:37:04.987592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:04.987565    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"
Oct 29 22:37:05.738517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:05.738479    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:07.281683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:07.281647    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:08.033448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:08.033412    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"]
Oct 29 22:37:08.047061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:08.047030    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f22ac5-hdjdb"]
Oct 29 22:37:08.116289 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:08.116249    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9175cc-da29-4c15-a2c3-a32b7196fd0c" path="/var/lib/kubelet/pods/5a9175cc-da29-4c15-a2c3-a32b7196fd0c/volumes"
Oct 29 22:37:08.944783 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:08.944748    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:10.709965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:10.709933    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:10.845439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:10.845406    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:11.681590 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:11.681560    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:12.421928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:12.421885    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:14.001001 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.000968    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"]
Oct 29 22:37:14.002016 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.001990    4860 cpu_manager.go:414] "containerMap: removing container" podUID="de0281ae-c434-4ebd-b8b3-bb6cf61693c3" containerName="revoke-keys"
Oct 29 22:37:14.002016 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.002016    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0281ae-c434-4ebd-b8b3-bb6cf61693c3" containerName="revoke-keys"
Oct 29 22:37:14.002236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.002220    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0281ae-c434-4ebd-b8b3-bb6cf61693c3" containerName="revoke-keys"
Oct 29 22:37:14.005338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.005310    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.027208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.027184    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:14.030101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.030063    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"]
Oct 29 22:37:14.148355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.148296    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.148690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.148662    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.148815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.148730    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.148990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.148974    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.149226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.149082    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.149226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.149136    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jn88\" (UniqueName: \"kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-kube-api-access-9jn88\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.149884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.149469    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.149884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.149503    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.149884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.149558    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.250655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.250615    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.250856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.250746    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.250856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.250782    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jn88\" (UniqueName: \"kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-kube-api-access-9jn88\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.250998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.250895    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.250998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.250919    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.250998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.250986    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.251207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.251045    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.251207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.251163    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.251356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.251233    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.251468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.251439    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.251637 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.251465    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.251637 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.251598    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.251779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.251758    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.251911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.251888    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.253300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.253274    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.253674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.253657    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.272002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.271970    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.272183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.272149    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jn88\" (UniqueName: \"kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-kube-api-access-9jn88\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-xvdlm\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.320102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.320077    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:14.642315 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:37:14.642287    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d54a31_5af8_4540_bb0d_36eaa9efa86f.slice/crio-1f8bcad83de8ca9a6fedb2b527a318fbc7a2955dc7068549eeb118acde5c7da3 WatchSource:0}: Error finding container 1f8bcad83de8ca9a6fedb2b527a318fbc7a2955dc7068549eeb118acde5c7da3: Status 404 returned error can't find the container with id 1f8bcad83de8ca9a6fedb2b527a318fbc7a2955dc7068549eeb118acde5c7da3
Oct 29 22:37:14.645758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.645720    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:14.645882 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.645810    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:14.645882 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.645856    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:14.648497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:14.648471    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"]
Oct 29 22:37:15.236136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:15.236097    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" event={"ID":"27d54a31-5af8-4540-bb0d-36eaa9efa86f","Type":"ContainerStarted","Data":"dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee"}
Oct 29 22:37:15.236615 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:15.236152    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" event={"ID":"27d54a31-5af8-4540-bb0d-36eaa9efa86f","Type":"ContainerStarted","Data":"1f8bcad83de8ca9a6fedb2b527a318fbc7a2955dc7068549eeb118acde5c7da3"}
Oct 29 22:37:15.295347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:15.295054    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" podStartSLOduration=2.295037568 podStartE2EDuration="2.295037568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:15.294377816 +0000 UTC m=+1377.319153964" watchObservedRunningTime="2026-10-29 22:37:15.295037568 +0000 UTC m=+1377.319813703"
Oct 29 22:37:15.320287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:15.320257    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:15.321610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:15.321586    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:37:15.321735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:15.321651    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:37:15.820961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:15.820923    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:16.320481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:16.320448    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:37:16.320889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:16.320499    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:37:17.323406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:17.320579    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:37:17.323406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:17.320667    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:37:17.456180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:17.456142    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:17.593208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:17.593112    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:17.705550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:17.705519    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:18.037521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:18.037483    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"]
Oct 29 22:37:18.050163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:18.050127    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-730de389-9vpvh"]
Oct 29 22:37:18.090277 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:37:18.090240    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:37:18.117726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:18.117695    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f370020-4408-4381-9edd-9ecfd8228ad8" path="/var/lib/kubelet/pods/8f370020-4408-4381-9edd-9ecfd8228ad8/volumes"
Oct 29 22:37:18.320970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:18.320880    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:37:18.321133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:18.320966    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:37:18.744012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:18.743089    4860 scope.go:122] "RemoveContainer" containerID="9d75ca73c938317ca706c9c3c05b31d7303f4d5bf5a631ae34d98d992c077769"
Oct 29 22:37:18.819200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:18.819152    4860 scope.go:122] "RemoveContainer" containerID="6bba72acaf32c4d8b78550164ef1aa631750ab06622b9b1644ba78d84cd06205"
Oct 29 22:37:18.831773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:18.831754    4860 scope.go:122] "RemoveContainer" containerID="debdd4c35749ba4d2b2ec7df27aea3db2b8fcb457476a1e36dc8d98eef6e377e"
Oct 29 22:37:18.842515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:18.842491    4860 scope.go:122] "RemoveContainer" containerID="8b03a95830a4b00e14fe2fe21b67224ab8d18cab2578749de2f88f830ef01630"
Oct 29 22:37:18.854991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:18.854972    4860 scope.go:122] "RemoveContainer" containerID="06d0fb205450f0f15bc7f1f4a2a272b94c46f614359431d8aa71f41123c316d2"
Oct 29 22:37:18.866046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:18.866022    4860 scope.go:122] "RemoveContainer" containerID="4cc47be3ab5839aac9601c5ee5e95acabb670c42c0bedddf8cefa0eef842689d"
Oct 29 22:37:18.905908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:18.905874    4860 scope.go:122] "RemoveContainer" containerID="6903242604c67f174c4ec5bb69594a24f3c2b47595d34080afed3cc54110919f"
Oct 29 22:37:19.042970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:19.042938    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"]
Oct 29 22:37:19.052937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:19.052906    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"]
Oct 29 22:37:19.062130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:19.062106    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dcfad3b-hkt4l"]
Oct 29 22:37:19.095347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:19.094744    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1d54390-27g4s"]
Oct 29 22:37:19.257587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:19.257501    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:19.320532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:19.320490    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:37:19.320714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:19.320542    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:37:20.113963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:20.113931    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe55b30-3f30-4718-99e3-7144caa7da9b" path="/var/lib/kubelet/pods/7fe55b30-3f30-4718-99e3-7144caa7da9b/volumes"
Oct 29 22:37:20.114464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:20.114445    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a71204-5e6e-4ee4-aae2-6daeabe95b7c" path="/var/lib/kubelet/pods/e2a71204-5e6e-4ee4-aae2-6daeabe95b7c/volumes"
Oct 29 22:37:20.320405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:20.320367    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:37:20.320576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:20.320432    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:37:21.007066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:21.007035    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:21.320508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:21.320418    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:37:21.320921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:21.320497    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:37:22.320602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:22.320573    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:37:22.320971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:22.320620    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:37:22.594762 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:22.594665    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:23.325592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:23.325562    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:23.326045 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:23.325928    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:23.326769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:23.326753    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:37:24.139234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:24.139165    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:25.711887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:25.711851    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:27.277646 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:27.277608    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:28.762105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:28.762070    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:29.046480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:29.046397    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"]
Oct 29 22:37:29.054387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:29.054356    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04c8a962-8899s"]
Oct 29 22:37:30.231457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.231226    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb3272a7-2bc8-4740-8de4-f34616108e39" path="/var/lib/kubelet/pods/fb3272a7-2bc8-4740-8de4-f34616108e39/volumes"
Oct 29 22:37:30.689141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.689095    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"]
Oct 29 22:37:30.690200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.690171    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:30.695943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.695924    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:30.710392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.710366    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"]
Oct 29 22:37:30.711989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.711955    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg\" (UID: \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:30.712241 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.712169    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg\" (UID: \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:30.712367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.712317    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455km\" (UniqueName: \"kubernetes.io/projected/06dd2821-5329-47e0-a877-6fd99dcc8b3e-kube-api-access-455km\") pod \"maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg\" (UID: \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:30.728386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.715383    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:37:30.728386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.716001    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:37:30.728386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.716079    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:37:30.728386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.726461    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:37:30.837346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.814673    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg\" (UID: \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:30.837675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.827885    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg\" (UID: \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:30.838599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.837800    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg\" (UID: \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:30.838599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.838049    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455km\" (UniqueName: \"kubernetes.io/projected/06dd2821-5329-47e0-a877-6fd99dcc8b3e-kube-api-access-455km\") pod \"maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg\" (UID: \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:30.838599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.838562    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg\" (UID: \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:30.863523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:30.863465    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-455km\" (UniqueName: \"kubernetes.io/projected/06dd2821-5329-47e0-a877-6fd99dcc8b3e-kube-api-access-455km\") pod \"maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg\" (UID: \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:31.017800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:31.017708    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:31.234510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:31.231381    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"]
Oct 29 22:37:31.309401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:31.308967    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25de7e1-x69zq"]
Oct 29 22:37:31.637131 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:37:31.637094    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06dd2821_5329_47e0_a877_6fd99dcc8b3e.slice/crio-797044a59868ae277a325b2e09b75d625e6e6299bcb24201d4713a482504e7d3 WatchSource:0}: Error finding container 797044a59868ae277a325b2e09b75d625e6e6299bcb24201d4713a482504e7d3: Status 404 returned error can't find the container with id 797044a59868ae277a325b2e09b75d625e6e6299bcb24201d4713a482504e7d3
Oct 29 22:37:31.639456 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:31.639431    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"]
Oct 29 22:37:32.147587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:32.147555    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f28939-d1c6-48db-b86b-01bea1e8a15b" path="/var/lib/kubelet/pods/e6f28939-d1c6-48db-b86b-01bea1e8a15b/volumes"
Oct 29 22:37:32.602727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:32.602684    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg" event={"ID":"06dd2821-5329-47e0-a877-6fd99dcc8b3e","Type":"ContainerStarted","Data":"f0073e6d711eb14b5faa607b88693a84631695fa03ded98cb6f7f6d2f1822350"}
Oct 29 22:37:32.626348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:32.606505    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg" event={"ID":"06dd2821-5329-47e0-a877-6fd99dcc8b3e","Type":"ContainerStarted","Data":"797044a59868ae277a325b2e09b75d625e6e6299bcb24201d4713a482504e7d3"}
Oct 29 22:37:32.708360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:32.705488    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg" podStartSLOduration=2.7054700289999998 podStartE2EDuration="2.705470029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:32.68773017 +0000 UTC m=+1394.712506327" watchObservedRunningTime="2026-10-29 22:37:32.705470029 +0000 UTC m=+1394.730246161"
Oct 29 22:37:33.061914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:33.061865    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"]
Oct 29 22:37:33.091500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:33.068798    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a7c2ff-mfztr"]
Oct 29 22:37:33.161773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:33.161311    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:33.654308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:33.654276    4860 generic.go:356] "Generic (PLEG): container finished" podID="06dd2821-5329-47e0-a877-6fd99dcc8b3e" containerID="f0073e6d711eb14b5faa607b88693a84631695fa03ded98cb6f7f6d2f1822350" exitCode=0
Oct 29 22:37:33.654811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:33.654362    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg" event={"ID":"06dd2821-5329-47e0-a877-6fd99dcc8b3e","Type":"ContainerDied","Data":"f0073e6d711eb14b5faa607b88693a84631695fa03ded98cb6f7f6d2f1822350"}
Oct 29 22:37:33.847685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:33.847583    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:34.139572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.139535    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ccb50ac-9a19-4084-a015-eafd58a59202" path="/var/lib/kubelet/pods/9ccb50ac-9a19-4084-a015-eafd58a59202/volumes"
Oct 29 22:37:34.579558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.579526    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:34.882361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.882319    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:34.972564 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.972532    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06dd2821-5329-47e0-a877-6fd99dcc8b3e-kube-api-access-455km\" (UniqueName: \"kubernetes.io/projected/06dd2821-5329-47e0-a877-6fd99dcc8b3e-kube-api-access-455km\") pod \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\" (UID: \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\") "
Oct 29 22:37:34.972758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.972602    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-xks-ca\") pod \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\" (UID: \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\") "
Oct 29 22:37:34.972758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.972666    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-openshift-ca\") pod \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\" (UID: \"06dd2821-5329-47e0-a877-6fd99dcc8b3e\") "
Oct 29 22:37:34.973760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.973733    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-openshift-ca" pod "06dd2821-5329-47e0-a877-6fd99dcc8b3e" (UID: "06dd2821-5329-47e0-a877-6fd99dcc8b3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:37:34.973930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.973910    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:37:34.975381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.975308    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06dd2821-5329-47e0-a877-6fd99dcc8b3e-kube-api-access-455km" pod "06dd2821-5329-47e0-a877-6fd99dcc8b3e" (UID: "06dd2821-5329-47e0-a877-6fd99dcc8b3e"). InnerVolumeSpecName "kube-api-access-455km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:37:34.975596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.975570    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-xks-ca" pod "06dd2821-5329-47e0-a877-6fd99dcc8b3e" (UID: "06dd2821-5329-47e0-a877-6fd99dcc8b3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:37:34.988981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.988895    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"]
Oct 29 22:37:34.992398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.990073    4860 cpu_manager.go:414] "containerMap: removing container" podUID="06dd2821-5329-47e0-a877-6fd99dcc8b3e" containerName="revoke-keys"
Oct 29 22:37:34.992398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.990094    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06dd2821-5329-47e0-a877-6fd99dcc8b3e" containerName="revoke-keys"
Oct 29 22:37:34.998971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:34.996079    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="06dd2821-5329-47e0-a877-6fd99dcc8b3e" containerName="revoke-keys"
Oct 29 22:37:35.000284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.000261    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.017787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.017755    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"]
Oct 29 22:37:35.019449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.018122    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-aeda8c-openshift-default-dockercfg-qhrz8\""
Oct 29 22:37:35.075603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.075561    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:35.076299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.076274    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-455km\" (UniqueName: \"kubernetes.io/projected/06dd2821-5329-47e0-a877-6fd99dcc8b3e-kube-api-access-455km\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:37:35.076447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.076433    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd2821-5329-47e0-a877-6fd99dcc8b3e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:37:35.179173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.178983    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-data\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.179490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.179464    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-socket\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.179853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.179834    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-credential-socket\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.180016 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.179997    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-podinfo\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.180113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.180045    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm6vw\" (UniqueName: \"kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-kube-api-access-vm6vw\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.180175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.180113    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istiod-ca-cert\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.180175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.180169    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-certs\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.180414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.180384    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-envoy\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.180563 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.180516    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-token\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.282684 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.282648    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-envoy\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.282854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.282697    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-token\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.282907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.282888    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-data\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.282977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.282957    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-socket\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.283025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.283003    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-credential-socket\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.283080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.283041    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-podinfo\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.283140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.283081    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm6vw\" (UniqueName: \"kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-kube-api-access-vm6vw\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.283203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.283147    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istiod-ca-cert\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.283311 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.283285    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-data\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.283400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.283359    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-certs\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.283463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.283417    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-socket\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.283535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.283514    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-credential-socket\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.283744 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.283724    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-certs\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.283936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.283896    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istiod-ca-cert\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.285743 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.285720    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-envoy\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.285865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.285753    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-podinfo\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.307432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.307373    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm6vw\" (UniqueName: \"kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-kube-api-access-vm6vw\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.313718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.313688    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-token\") pod \"e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") " pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.333752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.333627    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:35.373885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.373835    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:35.678337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.678290    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"
Oct 29 22:37:35.678533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.678364    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg" event={"ID":"06dd2821-5329-47e0-a877-6fd99dcc8b3e","Type":"ContainerDied","Data":"797044a59868ae277a325b2e09b75d625e6e6299bcb24201d4713a482504e7d3"}
Oct 29 22:37:35.678533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.678424    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="797044a59868ae277a325b2e09b75d625e6e6299bcb24201d4713a482504e7d3"
Oct 29 22:37:35.689832 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:37:35.689801    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d794f4f_1519_4aa9_8cd5_dc6016b7bdb7.slice/crio-12fd6c8798fc8d3fd591bb62fadf8951241f42a9071408bf0ea15c75bdf11d0e WatchSource:0}: Error finding container 12fd6c8798fc8d3fd591bb62fadf8951241f42a9071408bf0ea15c75bdf11d0e: Status 404 returned error can't find the container with id 12fd6c8798fc8d3fd591bb62fadf8951241f42a9071408bf0ea15c75bdf11d0e
Oct 29 22:37:35.692769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.692719    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:35.692860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.692818    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:35.692910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.692881    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:35.697381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:35.697357    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"]
Oct 29 22:37:36.689566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:36.689527    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" event={"ID":"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7","Type":"ContainerStarted","Data":"8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a"}
Oct 29 22:37:36.690100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:36.689596    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" event={"ID":"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7","Type":"ContainerStarted","Data":"12fd6c8798fc8d3fd591bb62fadf8951241f42a9071408bf0ea15c75bdf11d0e"}
Oct 29 22:37:36.765675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:36.765627    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" podStartSLOduration=2.765613918 podStartE2EDuration="2.765613918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:36.748375449 +0000 UTC m=+1398.773151603" watchObservedRunningTime="2026-10-29 22:37:36.765613918 +0000 UTC m=+1398.790390049"
Oct 29 22:37:36.771175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:36.771148    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:37.008132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:37.008053    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:37.333877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:37.333799    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:37.340070 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:37.340034    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:37.707374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:37.701435    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:37.707374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:37.702757    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:37:38.848391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:38.848310    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:39.157823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:39.157716    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:39.277490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:39.277453    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:40.645627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:40.645594    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:41.044695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:41.044662    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"]
Oct 29 22:37:41.112001 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:41.111969    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"]
Oct 29 22:37:41.215824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:41.215787    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:41.350804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:41.350726    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:42.446014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.445980    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:42.561876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.561841    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:42.600460 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.600420    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"]
Oct 29 22:37:42.605473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.605451    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"
Oct 29 22:37:42.631527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.626342    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"]
Oct 29 22:37:42.631527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.629344    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-697jr\" (UniqueName: \"kubernetes.io/projected/26b8069e-48b2-463d-9268-7ad8298922a7-kube-api-access-697jr\") pod \"payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n\" (UID: \"26b8069e-48b2-463d-9268-7ad8298922a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"
Oct 29 22:37:42.631527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.629523    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b8069e-48b2-463d-9268-7ad8298922a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n\" (UID: \"26b8069e-48b2-463d-9268-7ad8298922a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"
Oct 29 22:37:42.631527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.630072    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-aeda8c\""
Oct 29 22:37:42.631527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.630196    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-aeda8c-dockercfg-wqf6h\""
Oct 29 22:37:42.731857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.731763    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-697jr\" (UniqueName: \"kubernetes.io/projected/26b8069e-48b2-463d-9268-7ad8298922a7-kube-api-access-697jr\") pod \"payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n\" (UID: \"26b8069e-48b2-463d-9268-7ad8298922a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"
Oct 29 22:37:42.732037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.731949    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b8069e-48b2-463d-9268-7ad8298922a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n\" (UID: \"26b8069e-48b2-463d-9268-7ad8298922a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"
Oct 29 22:37:42.733104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.733073    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b8069e-48b2-463d-9268-7ad8298922a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n\" (UID: \"26b8069e-48b2-463d-9268-7ad8298922a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"
Oct 29 22:37:42.755073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.755043    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-697jr\" (UniqueName: \"kubernetes.io/projected/26b8069e-48b2-463d-9268-7ad8298922a7-kube-api-access-697jr\") pod \"payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n\" (UID: \"26b8069e-48b2-463d-9268-7ad8298922a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"
Oct 29 22:37:42.940500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:42.940465    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"
Oct 29 22:37:43.262351 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:37:43.262307    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b8069e_48b2_463d_9268_7ad8298922a7.slice/crio-b52ad140699e411a4e0a8e32fdfd8d03f1a99912588298dbd92fa52c85f11519 WatchSource:0}: Error finding container b52ad140699e411a4e0a8e32fdfd8d03f1a99912588298dbd92fa52c85f11519: Status 404 returned error can't find the container with id b52ad140699e411a4e0a8e32fdfd8d03f1a99912588298dbd92fa52c85f11519
Oct 29 22:37:43.265123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:43.265096    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"]
Oct 29 22:37:43.802619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:43.802585    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n" event={"ID":"26b8069e-48b2-463d-9268-7ad8298922a7","Type":"ContainerStarted","Data":"fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4"}
Oct 29 22:37:43.803088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:43.802643    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n" event={"ID":"26b8069e-48b2-463d-9268-7ad8298922a7","Type":"ContainerStarted","Data":"b52ad140699e411a4e0a8e32fdfd8d03f1a99912588298dbd92fa52c85f11519"}
Oct 29 22:37:43.803088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:43.802682    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"
Oct 29 22:37:43.842721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:43.842675    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n" podStartSLOduration=1.842663342 podStartE2EDuration="1.842663342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:43.842163116 +0000 UTC m=+1405.866939269" watchObservedRunningTime="2026-10-29 22:37:43.842663342 +0000 UTC m=+1405.867439532"
Oct 29 22:37:44.035504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.035475    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"]
Oct 29 22:37:44.043518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.043490    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b369ae-zs2kc"]
Oct 29 22:37:44.050043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.049997    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"]
Oct 29 22:37:44.061380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.061254    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8578faa4-kxmjg"]
Oct 29 22:37:44.129656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.129619    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6095b303-414b-4f09-94fe-99216332dff3" path="/var/lib/kubelet/pods/6095b303-414b-4f09-94fe-99216332dff3/volumes"
Oct 29 22:37:44.130159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.130144    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7def8e50-a52d-4771-9d3a-3cef50e4e5af" path="/var/lib/kubelet/pods/7def8e50-a52d-4771-9d3a-3cef50e4e5af/volumes"
Oct 29 22:37:44.272613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.272577    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:44.526705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.526677    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"]
Oct 29 22:37:44.532827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.532804    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:44.546575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.546552    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"]
Oct 29 22:37:44.554718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.554694    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:37:44.554857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.554740    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:37:44.554857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.554824    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:37:44.554857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.554845    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:37:44.628008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.627976    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7\" (UID: \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:44.628161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.628025    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7\" (UID: \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:44.628209 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.628157    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45rgb\" (UniqueName: \"kubernetes.io/projected/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-kube-api-access-45rgb\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7\" (UID: \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:44.729939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.729908    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7\" (UID: \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:44.730104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.729955    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7\" (UID: \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:44.730104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.730023    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rgb\" (UniqueName: \"kubernetes.io/projected/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-kube-api-access-45rgb\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7\" (UID: \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:44.730729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.730702    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7\" (UID: \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:44.732429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.732407    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7\" (UID: \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:44.751169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.751146    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45rgb\" (UniqueName: \"kubernetes.io/projected/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-kube-api-access-45rgb\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7\" (UID: \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:44.849219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:44.849151    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:45.153828 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:37:45.153797    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7dcccb8_b2e6_4e80_bb50_c0f0cf7c65d5.slice/crio-ff77a025f33cf1029aba213722d7a3a23c7034eb7eec04df0c2324e7004b5797 WatchSource:0}: Error finding container ff77a025f33cf1029aba213722d7a3a23c7034eb7eec04df0c2324e7004b5797: Status 404 returned error can't find the container with id ff77a025f33cf1029aba213722d7a3a23c7034eb7eec04df0c2324e7004b5797
Oct 29 22:37:45.157832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:45.157808    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"]
Oct 29 22:37:45.823624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:45.823586    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7" event={"ID":"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5","Type":"ContainerStarted","Data":"bdde5a5ce63bbdd750d40f8d2904a598a72157f4e04fe1f6c2e37a00d4c93858"}
Oct 29 22:37:45.823817 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:45.823645    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7" event={"ID":"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5","Type":"ContainerStarted","Data":"ff77a025f33cf1029aba213722d7a3a23c7034eb7eec04df0c2324e7004b5797"}
Oct 29 22:37:45.867401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:45.867296    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7" podStartSLOduration=1.867282697 podStartE2EDuration="1.867282697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:45.862711343 +0000 UTC m=+1407.887487497" watchObservedRunningTime="2026-10-29 22:37:45.867282697 +0000 UTC m=+1407.892058828"
Oct 29 22:37:46.044993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.044946    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5" podUID="02b3011f-d481-4687-851f-c170fe6ef09f" containerName="payload-processing" containerID="cri-o://6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974" gracePeriod=25
Oct 29 22:37:46.112548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.112497    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2" podUID="56d3b90a-1eb7-4180-80f4-61b0e7647078" containerName="payload-pre-processing" containerID="cri-o://c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe" gracePeriod=25
Oct 29 22:37:46.112702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.112656    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:46.434892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.434862    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:46.456661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.456603    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"
Oct 29 22:37:46.481149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.481122    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"
Oct 29 22:37:46.570775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.570739    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b3011f-d481-4687-851f-c170fe6ef09f-kube-api-access-w6hx9\" (UniqueName: \"kubernetes.io/projected/02b3011f-d481-4687-851f-c170fe6ef09f-kube-api-access-w6hx9\") pod \"02b3011f-d481-4687-851f-c170fe6ef09f\" (UID: \"02b3011f-d481-4687-851f-c170fe6ef09f\") "
Oct 29 22:37:46.571008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.570988    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02b3011f-d481-4687-851f-c170fe6ef09f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b3011f-d481-4687-851f-c170fe6ef09f-plugins-config-volume\") pod \"02b3011f-d481-4687-851f-c170fe6ef09f\" (UID: \"02b3011f-d481-4687-851f-c170fe6ef09f\") "
Oct 29 22:37:46.571416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.571388    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b3011f-d481-4687-851f-c170fe6ef09f-plugins-config-volume" pod "02b3011f-d481-4687-851f-c170fe6ef09f" (UID: "02b3011f-d481-4687-851f-c170fe6ef09f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:37:46.571950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.571929    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b3011f-d481-4687-851f-c170fe6ef09f-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:37:46.574254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.574226    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b3011f-d481-4687-851f-c170fe6ef09f-kube-api-access-w6hx9" pod "02b3011f-d481-4687-851f-c170fe6ef09f" (UID: "02b3011f-d481-4687-851f-c170fe6ef09f"). InnerVolumeSpecName "kube-api-access-w6hx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:37:46.673081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.673050    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d3b90a-1eb7-4180-80f4-61b0e7647078-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d3b90a-1eb7-4180-80f4-61b0e7647078-plugins-config-volume\") pod \"56d3b90a-1eb7-4180-80f4-61b0e7647078\" (UID: \"56d3b90a-1eb7-4180-80f4-61b0e7647078\") "
Oct 29 22:37:46.673486 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.673459    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d3b90a-1eb7-4180-80f4-61b0e7647078-plugins-config-volume" pod "56d3b90a-1eb7-4180-80f4-61b0e7647078" (UID: "56d3b90a-1eb7-4180-80f4-61b0e7647078"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:37:46.673597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.673494    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d3b90a-1eb7-4180-80f4-61b0e7647078-kube-api-access-jckjd\" (UniqueName: \"kubernetes.io/projected/56d3b90a-1eb7-4180-80f4-61b0e7647078-kube-api-access-jckjd\") pod \"56d3b90a-1eb7-4180-80f4-61b0e7647078\" (UID: \"56d3b90a-1eb7-4180-80f4-61b0e7647078\") "
Oct 29 22:37:46.674129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.674107    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6hx9\" (UniqueName: \"kubernetes.io/projected/02b3011f-d481-4687-851f-c170fe6ef09f-kube-api-access-w6hx9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:37:46.674205 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.674133    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d3b90a-1eb7-4180-80f4-61b0e7647078-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:37:46.675583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.675538    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d3b90a-1eb7-4180-80f4-61b0e7647078-kube-api-access-jckjd" pod "56d3b90a-1eb7-4180-80f4-61b0e7647078" (UID: "56d3b90a-1eb7-4180-80f4-61b0e7647078"). InnerVolumeSpecName "kube-api-access-jckjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:37:46.775972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.775943    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jckjd\" (UniqueName: \"kubernetes.io/projected/56d3b90a-1eb7-4180-80f4-61b0e7647078-kube-api-access-jckjd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:37:46.834947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.834920    4860 generic.go:356] "Generic (PLEG): container finished" podID="02b3011f-d481-4687-851f-c170fe6ef09f" containerID="6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974" exitCode=0
Oct 29 22:37:46.835060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.834975    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"
Oct 29 22:37:46.835060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.835000    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5" event={"ID":"02b3011f-d481-4687-851f-c170fe6ef09f","Type":"ContainerDied","Data":"6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974"}
Oct 29 22:37:46.835060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.835051    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5" event={"ID":"02b3011f-d481-4687-851f-c170fe6ef09f","Type":"ContainerDied","Data":"a5cca5deb7a81192d7b3957d23581c81326b1eeb8f892bb1b40c31f809408b51"}
Oct 29 22:37:46.835219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.835074    4860 scope.go:122] "RemoveContainer" containerID="6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974"
Oct 29 22:37:46.836678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.836654    4860 generic.go:356] "Generic (PLEG): container finished" podID="56d3b90a-1eb7-4180-80f4-61b0e7647078" containerID="c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe" exitCode=0
Oct 29 22:37:46.836790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.836685    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2" event={"ID":"56d3b90a-1eb7-4180-80f4-61b0e7647078","Type":"ContainerDied","Data":"c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe"}
Oct 29 22:37:46.836790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.836701    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"
Oct 29 22:37:46.836790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.836737    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2" event={"ID":"56d3b90a-1eb7-4180-80f4-61b0e7647078","Type":"ContainerDied","Data":"3d6213cd5d45e8d4e19b853af5871a43299773291a602f2eb67ce66a269c02fd"}
Oct 29 22:37:46.839077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.839058    4860 generic.go:356] "Generic (PLEG): container finished" podID="e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5" containerID="bdde5a5ce63bbdd750d40f8d2904a598a72157f4e04fe1f6c2e37a00d4c93858" exitCode=0
Oct 29 22:37:46.839189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.839157    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7" event={"ID":"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5","Type":"ContainerDied","Data":"bdde5a5ce63bbdd750d40f8d2904a598a72157f4e04fe1f6c2e37a00d4c93858"}
Oct 29 22:37:46.848202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.848182    4860 scope.go:122] "RemoveContainer" containerID="6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974"
Oct 29 22:37:46.848470 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:37:46.848452    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974\": container with ID starting with 6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974 not found: ID does not exist" containerID="6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974"
Oct 29 22:37:46.848529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.848478    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974"} err="failed to get container status \"6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974\": rpc error: code = NotFound desc = could not find container \"6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974\": container with ID starting with 6adb6092d90067992cb0b3cce826adf4fcf5bb7774dcfb82449426d9dc55b974 not found: ID does not exist"
Oct 29 22:37:46.848529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.848497    4860 scope.go:122] "RemoveContainer" containerID="c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe"
Oct 29 22:37:46.859166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.859149    4860 scope.go:122] "RemoveContainer" containerID="c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe"
Oct 29 22:37:46.859467 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:37:46.859441    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe\": container with ID starting with c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe not found: ID does not exist" containerID="c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe"
Oct 29 22:37:46.859546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.859465    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe"} err="failed to get container status \"c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe\": rpc error: code = NotFound desc = could not find container \"c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe\": container with ID starting with c3d057c661f46584981a52f8a151833787a6998f9d492070f55e4928b249d5fe not found: ID does not exist"
Oct 29 22:37:46.938572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.938546    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"]
Oct 29 22:37:46.946743 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.946717    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1131a29d-79fbb5fd4c-rqqk2"]
Oct 29 22:37:46.980244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.980218    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"]
Oct 29 22:37:46.990784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:46.990754    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1131a29d-6d7ccc5c4d-mdml5"]
Oct 29 22:37:47.037634 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.037597    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"]
Oct 29 22:37:47.050064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.050034    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d81d613-2sh9x"]
Oct 29 22:37:47.119988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.119961    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:47.555274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.555242    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"]
Oct 29 22:37:47.556041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.556023    4860 cpu_manager.go:414] "containerMap: removing container" podUID="02b3011f-d481-4687-851f-c170fe6ef09f" containerName="payload-processing"
Oct 29 22:37:47.556096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.556042    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b3011f-d481-4687-851f-c170fe6ef09f" containerName="payload-processing"
Oct 29 22:37:47.556096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.556072    4860 cpu_manager.go:414] "containerMap: removing container" podUID="56d3b90a-1eb7-4180-80f4-61b0e7647078" containerName="payload-pre-processing"
Oct 29 22:37:47.556096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.556077    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d3b90a-1eb7-4180-80f4-61b0e7647078" containerName="payload-pre-processing"
Oct 29 22:37:47.556235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.556225    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d3b90a-1eb7-4180-80f4-61b0e7647078" containerName="payload-pre-processing"
Oct 29 22:37:47.556271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.556247    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b3011f-d481-4687-851f-c170fe6ef09f" containerName="payload-processing"
Oct 29 22:37:47.561658 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.561638    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.589517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.589485    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"]
Oct 29 22:37:47.591341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.591305    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-39322f-openshift-default-dockercfg-hld6q\""
Oct 29 22:37:47.643646 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.643615    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:47.690484 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.690458    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-podinfo\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.690642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.690495    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-token\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.690642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.690582    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-credential-socket\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.690642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.690639    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-socket\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.690754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.690713    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dl76\" (UniqueName: \"kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-kube-api-access-7dl76\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.690822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.690809    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istiod-ca-cert\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.690858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.690844    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-certs\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.691040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.691025    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-data\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.691074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.691063    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-envoy\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.792681    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.793680    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-data\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.793733    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-envoy\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.793797    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-podinfo\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.793824    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-token\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.793866    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-credential-socket\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.793920    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-socket\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.793951    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dl76\" (UniqueName: \"kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-kube-api-access-7dl76\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.794029    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istiod-ca-cert\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.794067    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-certs\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.794555    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-certs\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.794779    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-data\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.795466    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-credential-socket\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.796570    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-socket\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.797218    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istiod-ca-cert\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.800214    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-envoy\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.804366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.802739    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-podinfo\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.860387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.859530    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dl76\" (UniqueName: \"kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-kube-api-access-7dl76\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.876346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.871152    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-token\") pod \"e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") " pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.889713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.889686    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:47.965148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:47.961117    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:48.166884 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:37:48.165942    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:37:48.168863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.168831    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b3011f-d481-4687-851f-c170fe6ef09f" path="/var/lib/kubelet/pods/02b3011f-d481-4687-851f-c170fe6ef09f/volumes"
Oct 29 22:37:48.169664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.169643    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d3b90a-1eb7-4180-80f4-61b0e7647078" path="/var/lib/kubelet/pods/56d3b90a-1eb7-4180-80f4-61b0e7647078/volumes"
Oct 29 22:37:48.170388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.170310    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6266e819-5a4a-4dd2-a11c-b008142baf37" path="/var/lib/kubelet/pods/6266e819-5a4a-4dd2-a11c-b008142baf37/volumes"
Oct 29 22:37:48.319174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.319143    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:48.347912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.347869    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-kube-api-access-45rgb\" (UniqueName: \"kubernetes.io/projected/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-kube-api-access-45rgb\") pod \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\" (UID: \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\") "
Oct 29 22:37:48.348090 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.347968    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-xks-ca\") pod \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\" (UID: \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\") "
Oct 29 22:37:48.352073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.350678    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-xks-ca" pod "e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5" (UID: "e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:37:48.352657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.352625    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-kube-api-access-45rgb" pod "e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5" (UID: "e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5"). InnerVolumeSpecName "kube-api-access-45rgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:37:48.449858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.449811    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-openshift-ca\") pod \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\" (UID: \"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5\") "
Oct 29 22:37:48.450233 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.450205    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-openshift-ca" pod "e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5" (UID: "e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:37:48.450776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.450755    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:37:48.450847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.450783    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:37:48.450847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.450795    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45rgb\" (UniqueName: \"kubernetes.io/projected/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5-kube-api-access-45rgb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:37:48.486527 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:37:48.486500    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a537d76_6b6c_4ae9_a18d_1556b9d26625.slice/crio-9ec9bbf8f32cd7e1195b713cbcdf4621d4e92ad100e87fb4b8472c61f02a5a9b WatchSource:0}: Error finding container 9ec9bbf8f32cd7e1195b713cbcdf4621d4e92ad100e87fb4b8472c61f02a5a9b: Status 404 returned error can't find the container with id 9ec9bbf8f32cd7e1195b713cbcdf4621d4e92ad100e87fb4b8472c61f02a5a9b
Oct 29 22:37:48.489119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.489094    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"]
Oct 29 22:37:48.489678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.489643    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:48.489759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.489725    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:48.489828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.489771    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:48.573366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.573333    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"]
Oct 29 22:37:48.574518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.574498    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5" containerName="revoke-keys"
Oct 29 22:37:48.574518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.574519    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5" containerName="revoke-keys"
Oct 29 22:37:48.574712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.574700    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5" containerName="revoke-keys"
Oct 29 22:37:48.579189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.579167    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.595189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.595165    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"]
Oct 29 22:37:48.595573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.595551    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cae55b-openshift-default-dockercfg-pxrq8\""
Oct 29 22:37:48.654230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.653918    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istiod-ca-cert\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.654230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.654013    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-data\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.654230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.654135    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-credential-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.654230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.654172    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-token\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.654230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.654197    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-envoy\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.654693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.654287    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-certs\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.654693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.654470    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jmw\" (UniqueName: \"kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-kube-api-access-d7jmw\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.654693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.654541    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-podinfo\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.654693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.654590    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.760427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.760312    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jmw\" (UniqueName: \"kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-kube-api-access-d7jmw\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.760427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.760384    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-podinfo\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.760427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.760409    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.760730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.760506    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istiod-ca-cert\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.760730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.760545    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-data\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.760730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.760654    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-credential-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.760730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.760692    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-token\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.760730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.760720    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-envoy\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.761065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.760792    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-certs\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.761065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.760933    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.761065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.760969    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-data\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.761228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.761133    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-certs\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.761367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.761299    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-credential-socket\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.761473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.761388    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istiod-ca-cert\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.763294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.763267    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-podinfo\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.763508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.763489    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-envoy\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.781427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.781404    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-token\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.781427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.781421    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jmw\" (UniqueName: \"kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-kube-api-access-d7jmw\") pod \"e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") " pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.890609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.890578    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"
Oct 29 22:37:48.890733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.890583    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7" event={"ID":"e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5","Type":"ContainerDied","Data":"ff77a025f33cf1029aba213722d7a3a23c7034eb7eec04df0c2324e7004b5797"}
Oct 29 22:37:48.890798 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.890743    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff77a025f33cf1029aba213722d7a3a23c7034eb7eec04df0c2324e7004b5797"
Oct 29 22:37:48.892797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.892776    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" event={"ID":"7a537d76-6b6c-4ae9-a18d-1556b9d26625","Type":"ContainerStarted","Data":"86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493"}
Oct 29 22:37:48.892911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.892814    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" event={"ID":"7a537d76-6b6c-4ae9-a18d-1556b9d26625","Type":"ContainerStarted","Data":"9ec9bbf8f32cd7e1195b713cbcdf4621d4e92ad100e87fb4b8472c61f02a5a9b"}
Oct 29 22:37:48.899345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.899307    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:48.942946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:48.942871    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" podStartSLOduration=1.9428512580000001 podStartE2EDuration="1.942851258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:48.938172603 +0000 UTC m=+1410.962948734" watchObservedRunningTime="2026-10-29 22:37:48.942851258 +0000 UTC m=+1410.967627390"
Oct 29 22:37:49.262943 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:37:49.262872    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70e8c56_6765_4caa_8e1d_3eb08b7777d5.slice/crio-46ff67258a04c8481bbcaa46d17df89a39fe5376c4ed0fc52066de953c537afd WatchSource:0}: Error finding container 46ff67258a04c8481bbcaa46d17df89a39fe5376c4ed0fc52066de953c537afd: Status 404 returned error can't find the container with id 46ff67258a04c8481bbcaa46d17df89a39fe5376c4ed0fc52066de953c537afd
Oct 29 22:37:49.266317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:49.266279    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:49.266516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:49.266420    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:49.266516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:49.266476    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:49.269113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:49.269089    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"]
Oct 29 22:37:49.890834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:49.890800    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:49.891703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:49.891672    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:49.896913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:49.896889    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:49.905157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:49.905131    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx" event={"ID":"b70e8c56-6765-4caa-8e1d-3eb08b7777d5","Type":"ContainerStarted","Data":"27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5"}
Oct 29 22:37:49.905287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:49.905175    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx" event={"ID":"b70e8c56-6765-4caa-8e1d-3eb08b7777d5","Type":"ContainerStarted","Data":"46ff67258a04c8481bbcaa46d17df89a39fe5376c4ed0fc52066de953c537afd"}
Oct 29 22:37:49.905419 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:49.905408    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:49.906453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:49.906431    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:37:50.899680 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:50.899644    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:50.901373 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:50.901307    4860 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 29 22:37:50.901513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:50.901390    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx" podUID="b70e8c56-6765-4caa-8e1d-3eb08b7777d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 29 22:37:51.335032 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:51.334994    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:51.460363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:51.459903    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:51.765312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:51.765278    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:51.903350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:51.902703    4860 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 29 22:37:51.903350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:51.902766    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx" podUID="b70e8c56-6765-4caa-8e1d-3eb08b7777d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 29 22:37:52.154079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:52.153992    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:53.049351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:53.042197    4860 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 29 22:37:53.049351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:53.042257    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx" podUID="b70e8c56-6765-4caa-8e1d-3eb08b7777d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 29 22:37:53.049351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:53.045709    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:53.398060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:53.397978    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:53.531759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:53.531725    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:53.650998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:53.650903    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:53.899823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:53.899778    4860 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 29 22:37:53.900079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:53.900055    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx" podUID="b70e8c56-6765-4caa-8e1d-3eb08b7777d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 29 22:37:54.130354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.129670    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx" podStartSLOduration=6.129631411 podStartE2EDuration="6.129631411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:50.043679921 +0000 UTC m=+1412.068456055" watchObservedRunningTime="2026-10-29 22:37:54.129631411 +0000 UTC m=+1416.154407546"
Oct 29 22:37:54.234354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.217060    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"]
Oct 29 22:37:54.283349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.275727    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"]
Oct 29 22:37:54.351489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.351457    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.365357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.363724    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"]
Oct 29 22:37:54.365357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.363779    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"]
Oct 29 22:37:54.365357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.363879    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.365986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.365951    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-data\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.366481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.366456    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-credential-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.366732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.366709    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.366985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.366968    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-envoy\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.367184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.367168    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-token\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.367374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.367359    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-certs\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.377618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.377529    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b8f3bc-6cc7-44e3-b406-d278376a586a-istiod-ca-cert\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.377954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.377931    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkxpr\" (UniqueName: \"kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-kube-api-access-xkxpr\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.378134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.378115    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-podinfo\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.386937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.386292    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1131a29d-openshift-default-dockercfg-8jjv9\""
Oct 29 22:37:54.494445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.493662    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-envoy\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.494445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.493741    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-token\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.494445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.494062    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-certs\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.494445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.494163    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-credential-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.494445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.494215    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b8f3bc-6cc7-44e3-b406-d278376a586a-istiod-ca-cert\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.494445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.494273    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-token\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.494445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.494344    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9fabed-9589-46a6-843a-6fd99d1278ff-istiod-ca-cert\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.495499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.495443    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b8f3bc-6cc7-44e3-b406-d278376a586a-istiod-ca-cert\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.495923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.495711    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-certs\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.495923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.495777    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw84v\" (UniqueName: \"kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-kube-api-access-dw84v\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.495923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.495830    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkxpr\" (UniqueName: \"kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-kube-api-access-xkxpr\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.495923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.495864    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-podinfo\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.497166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.496840    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-data\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.497166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.496907    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-data\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.497166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.496940    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-certs\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.497166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.496980    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.497166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.497069    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-podinfo\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.497166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.497131    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-envoy\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.497581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.497212    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-credential-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.497581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.497243    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.497765 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.497744    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-credential-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.498008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.497984    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.499750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.499685    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-data\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.499950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.499929    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-envoy\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.501861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.501837    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-podinfo\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.514085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.514063    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-token\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.517529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.517511    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkxpr\" (UniqueName: \"kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-kube-api-access-xkxpr\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.599710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.599675    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-token\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.599883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.599733    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9fabed-9589-46a6-843a-6fd99d1278ff-istiod-ca-cert\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.599883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.599865    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw84v\" (UniqueName: \"kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-kube-api-access-dw84v\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.600198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.600174    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-data\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.600289 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.600246    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-certs\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.600405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.600291    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.600482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.600402    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-podinfo\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.600482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.600472    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-envoy\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.600635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.600612    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9fabed-9589-46a6-843a-6fd99d1278ff-istiod-ca-cert\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.600695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.600653    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-data\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.600784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.600763    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-credential-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.600919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.600899    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-certs\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.601154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.601122    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.601263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.601178    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-credential-socket\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.602988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.602965    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-envoy\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.603782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.603760    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-podinfo\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.620756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.620733    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-token\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.620884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.620867    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw84v\" (UniqueName: \"kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-kube-api-access-dw84v\") pod \"e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") " pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.666659 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.666601    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:54.727855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.727495    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:54.728192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.728080    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:54.817390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.816578    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"
Oct 29 22:37:54.919573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.919085    4860 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 29 22:37:54.919573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:54.919142    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx" podUID="b70e8c56-6765-4caa-8e1d-3eb08b7777d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 29 22:37:55.133731 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:37:55.133697    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b8f3bc_6cc7_44e3_b406_d278376a586a.slice/crio-7e98b95f8290de4121b41fe0049635f488d6a0a6ca71b0b2635602516d44b2e8 WatchSource:0}: Error finding container 7e98b95f8290de4121b41fe0049635f488d6a0a6ca71b0b2635602516d44b2e8: Status 404 returned error can't find the container with id 7e98b95f8290de4121b41fe0049635f488d6a0a6ca71b0b2635602516d44b2e8
Oct 29 22:37:55.137171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.137134    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:55.137267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.137201    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:55.137267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.137232    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:55.137649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.137623    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"]
Oct 29 22:37:55.215120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.214408    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"]
Oct 29 22:37:55.221186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.221157    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" event={"ID":"05b8f3bc-6cc7-44e3-b406-d278376a586a","Type":"ContainerStarted","Data":"7e98b95f8290de4121b41fe0049635f488d6a0a6ca71b0b2635602516d44b2e8"}
Oct 29 22:37:55.222039 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:37:55.222016    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9fabed_9589_46a6_843a_6fd99d1278ff.slice/crio-9a01e72c3badbfe9fd01d93b061f27eefbbf29ebeee873569095452fea5967d3 WatchSource:0}: Error finding container 9a01e72c3badbfe9fd01d93b061f27eefbbf29ebeee873569095452fea5967d3: Status 404 returned error can't find the container with id 9a01e72c3badbfe9fd01d93b061f27eefbbf29ebeee873569095452fea5967d3
Oct 29 22:37:55.226087 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.226050    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:55.226198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.226123    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:55.226198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.226152    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:37:55.247070 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.247034    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:55.359264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.359197    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:55.924347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.922142    4860 patch_prober.go:29] interesting pod/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 29 22:37:55.924347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.922201    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx" podUID="b70e8c56-6765-4caa-8e1d-3eb08b7777d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 29 22:37:55.970347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:55.959466    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"]
Oct 29 22:37:56.079160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.077380    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:37:56.097226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.097200    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:37:56.098156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.098131    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:37:56.110407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.101110    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:37:56.110729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.110708    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:37:56.138858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.138827    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"]
Oct 29 22:37:56.218265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.218215    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2866h\" (UniqueName: \"kubernetes.io/projected/f8da5d76-1168-40e9-97ae-b320a9b84c76-kube-api-access-2866h\") pod \"maas-api-revoke-sub-e8b0d2c5-f5l72\" (UID: \"f8da5d76-1168-40e9-97ae-b320a9b84c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:37:56.218529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.218474    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b0d2c5-f5l72\" (UID: \"f8da5d76-1168-40e9-97ae-b320a9b84c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:37:56.218605 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.218577    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b0d2c5-f5l72\" (UID: \"f8da5d76-1168-40e9-97ae-b320a9b84c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:37:56.254747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.254707    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" event={"ID":"cd9fabed-9589-46a6-843a-6fd99d1278ff","Type":"ContainerStarted","Data":"8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366"}
Oct 29 22:37:56.254930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.254771    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" event={"ID":"cd9fabed-9589-46a6-843a-6fd99d1278ff","Type":"ContainerStarted","Data":"9a01e72c3badbfe9fd01d93b061f27eefbbf29ebeee873569095452fea5967d3"}
Oct 29 22:37:56.258008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.257978    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" event={"ID":"05b8f3bc-6cc7-44e3-b406-d278376a586a","Type":"ContainerStarted","Data":"84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164"}
Oct 29 22:37:56.319346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.315014    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" podStartSLOduration=2.314994321 podStartE2EDuration="2.314994321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:56.31300748 +0000 UTC m=+1418.337783611" watchObservedRunningTime="2026-10-29 22:37:56.314994321 +0000 UTC m=+1418.339770454"
Oct 29 22:37:56.320839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.320800    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b0d2c5-f5l72\" (UID: \"f8da5d76-1168-40e9-97ae-b320a9b84c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:37:56.321063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.321044    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b0d2c5-f5l72\" (UID: \"f8da5d76-1168-40e9-97ae-b320a9b84c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:37:56.329427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.323394    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b0d2c5-f5l72\" (UID: \"f8da5d76-1168-40e9-97ae-b320a9b84c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:37:56.329427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.324579    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2866h\" (UniqueName: \"kubernetes.io/projected/f8da5d76-1168-40e9-97ae-b320a9b84c76-kube-api-access-2866h\") pod \"maas-api-revoke-sub-e8b0d2c5-f5l72\" (UID: \"f8da5d76-1168-40e9-97ae-b320a9b84c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:37:56.331949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.331210    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b0d2c5-f5l72\" (UID: \"f8da5d76-1168-40e9-97ae-b320a9b84c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:37:56.371414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.367572    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2866h\" (UniqueName: \"kubernetes.io/projected/f8da5d76-1168-40e9-97ae-b320a9b84c76-kube-api-access-2866h\") pod \"maas-api-revoke-sub-e8b0d2c5-f5l72\" (UID: \"f8da5d76-1168-40e9-97ae-b320a9b84c76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:37:56.427596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.427529    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" podStartSLOduration=2.42750813 podStartE2EDuration="2.42750813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:56.401813379 +0000 UTC m=+1418.426589511" watchObservedRunningTime="2026-10-29 22:37:56.42750813 +0000 UTC m=+1418.452284263"
Oct 29 22:37:56.447861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.447383    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:37:56.611344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.602698    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:56.757352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.756800    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:37:56.758210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.758185    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:37:56.828430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.826498    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:37:56.828430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.826561    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" podUID="cd9fabed-9589-46a6-843a-6fd99d1278ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 29 22:37:56.828430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.826815    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:37:56.828430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.826857    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 29 22:37:56.942638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.940167    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:56.943185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.942842    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:56.949846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:56.949820    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:37:57.423003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:57.422968    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"]
Oct 29 22:37:57.424585 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:37:57.424445    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8da5d76_1168_40e9_97ae_b320a9b84c76.slice/crio-afab7e868010125b47870c704a7e503494d75f4fb2f323b9d4a544c050b101f0 WatchSource:0}: Error finding container afab7e868010125b47870c704a7e503494d75f4fb2f323b9d4a544c050b101f0: Status 404 returned error can't find the container with id afab7e868010125b47870c704a7e503494d75f4fb2f323b9d4a544c050b101f0
Oct 29 22:37:57.730360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:57.729253    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:37:57.730360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:57.729353    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" podUID="cd9fabed-9589-46a6-843a-6fd99d1278ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 29 22:37:57.743379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:57.742177    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:37:57.743379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:57.742251    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 29 22:37:58.037144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.037118    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"]
Oct 29 22:37:58.044869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.044844    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6b90f5-8jcs2"]
Oct 29 22:37:58.139222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.139186    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d114b79a-fab3-42f2-afe2-254d781491de" path="/var/lib/kubelet/pods/d114b79a-fab3-42f2-afe2-254d781491de/volumes"
Oct 29 22:37:58.158791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.158768    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"]
Oct 29 22:37:58.163577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.163561    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:37:58.234546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.234512    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"]
Oct 29 22:37:58.260587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.260559    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9b0e545-smzcb\" (UID: \"d1818f73-498f-47fd-afab-467d76cf7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:37:58.260713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.260637    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9b0e545-smzcb\" (UID: \"d1818f73-498f-47fd-afab-467d76cf7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:37:58.260713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.260669    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkr8n\" (UniqueName: \"kubernetes.io/projected/d1818f73-498f-47fd-afab-467d76cf7209-kube-api-access-dkr8n\") pod \"maas-api-revoke-sub-e9b0e545-smzcb\" (UID: \"d1818f73-498f-47fd-afab-467d76cf7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:37:58.344222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.344137    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72" event={"ID":"f8da5d76-1168-40e9-97ae-b320a9b84c76","Type":"ContainerStarted","Data":"5b460a85fd0cb4276ed7231b682c84162acbe92a05d7fbf4aca1b027f31a9f5c"}
Oct 29 22:37:58.344222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.344200    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72" event={"ID":"f8da5d76-1168-40e9-97ae-b320a9b84c76","Type":"ContainerStarted","Data":"afab7e868010125b47870c704a7e503494d75f4fb2f323b9d4a544c050b101f0"}
Oct 29 22:37:58.352522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.352488    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:58.362660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.362632    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9b0e545-smzcb\" (UID: \"d1818f73-498f-47fd-afab-467d76cf7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:37:58.362788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.362735    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9b0e545-smzcb\" (UID: \"d1818f73-498f-47fd-afab-467d76cf7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:37:58.362867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.362796    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkr8n\" (UniqueName: \"kubernetes.io/projected/d1818f73-498f-47fd-afab-467d76cf7209-kube-api-access-dkr8n\") pod \"maas-api-revoke-sub-e9b0e545-smzcb\" (UID: \"d1818f73-498f-47fd-afab-467d76cf7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:37:58.363273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.363246    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9b0e545-smzcb\" (UID: \"d1818f73-498f-47fd-afab-467d76cf7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:37:58.365450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.365424    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9b0e545-smzcb\" (UID: \"d1818f73-498f-47fd-afab-467d76cf7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:37:58.423081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.423045    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkr8n\" (UniqueName: \"kubernetes.io/projected/d1818f73-498f-47fd-afab-467d76cf7209-kube-api-access-dkr8n\") pod \"maas-api-revoke-sub-e9b0e545-smzcb\" (UID: \"d1818f73-498f-47fd-afab-467d76cf7209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:37:58.425611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.425558    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72" podStartSLOduration=3.4255386 podStartE2EDuration="3.4255386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:58.405593847 +0000 UTC m=+1420.430369999" watchObservedRunningTime="2026-10-29 22:37:58.4255386 +0000 UTC m=+1420.450314737"
Oct 29 22:37:58.502348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.498771    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:58.538449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.538415    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:37:58.651965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.651849    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:37:58.809351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.797575    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:37:58.809351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.797637    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 29 22:37:58.809351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.806456    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:37:58.809351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:58.806515    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" podUID="cd9fabed-9589-46a6-843a-6fd99d1278ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 29 22:37:59.438130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:59.434369    4860 generic.go:356] "Generic (PLEG): container finished" podID="f8da5d76-1168-40e9-97ae-b320a9b84c76" containerID="5b460a85fd0cb4276ed7231b682c84162acbe92a05d7fbf4aca1b027f31a9f5c" exitCode=0
Oct 29 22:37:59.438130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:59.434597    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72" event={"ID":"f8da5d76-1168-40e9-97ae-b320a9b84c76","Type":"ContainerDied","Data":"5b460a85fd0cb4276ed7231b682c84162acbe92a05d7fbf4aca1b027f31a9f5c"}
Oct 29 22:37:59.551299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:59.550848    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"]
Oct 29 22:37:59.728671 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:59.728584    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:37:59.728671 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:59.728640    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 29 22:37:59.728894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:59.728675    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:37:59.728894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:37:59.728713    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" podUID="cd9fabed-9589-46a6-843a-6fd99d1278ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 29 22:38:00.165231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:00.165198    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:00.286457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:00.282130    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:00.767350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:00.747451    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:38:00.767350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:00.747516    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 29 22:38:00.767350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:00.747967    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:38:00.767350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:00.748007    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" podUID="cd9fabed-9589-46a6-843a-6fd99d1278ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 29 22:38:00.812447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:00.810216    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:00.876476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:00.874110    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb" event={"ID":"d1818f73-498f-47fd-afab-467d76cf7209","Type":"ContainerStarted","Data":"f11630547b811653b0184291134d22b10542abbc3713dfc44d12d654e16e0253"}
Oct 29 22:38:00.876476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:00.874169    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb" event={"ID":"d1818f73-498f-47fd-afab-467d76cf7209","Type":"ContainerStarted","Data":"19ec72c74a06c422e4521b478aec01594350dd5786cd01bbd7f2ae889087ba92"}
Oct 29 22:38:01.039997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:00.968909    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb" podStartSLOduration=2.968888706 podStartE2EDuration="2.968888706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:00.954444895 +0000 UTC m=+1422.979221044" watchObservedRunningTime="2026-10-29 22:38:00.968888706 +0000 UTC m=+1422.993664839"
Oct 29 22:38:01.622386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.622260    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:38:01.729462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.729428    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:38:01.729644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.729482    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" podUID="cd9fabed-9589-46a6-843a-6fd99d1278ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 29 22:38:01.729644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.729629    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:38:01.729782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.729664    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 29 22:38:01.751044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.751014    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8da5d76-1168-40e9-97ae-b320a9b84c76-kube-api-access-2866h\" (UniqueName: \"kubernetes.io/projected/f8da5d76-1168-40e9-97ae-b320a9b84c76-kube-api-access-2866h\") pod \"f8da5d76-1168-40e9-97ae-b320a9b84c76\" (UID: \"f8da5d76-1168-40e9-97ae-b320a9b84c76\") "
Oct 29 22:38:01.751201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.751053    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-xks-ca\") pod \"f8da5d76-1168-40e9-97ae-b320a9b84c76\" (UID: \"f8da5d76-1168-40e9-97ae-b320a9b84c76\") "
Oct 29 22:38:01.751492 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.751462    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-openshift-ca\") pod \"f8da5d76-1168-40e9-97ae-b320a9b84c76\" (UID: \"f8da5d76-1168-40e9-97ae-b320a9b84c76\") "
Oct 29 22:38:01.751949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.751923    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-openshift-ca" pod "f8da5d76-1168-40e9-97ae-b320a9b84c76" (UID: "f8da5d76-1168-40e9-97ae-b320a9b84c76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:01.752461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.752438    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:01.753521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.753492    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8da5d76-1168-40e9-97ae-b320a9b84c76-kube-api-access-2866h" pod "f8da5d76-1168-40e9-97ae-b320a9b84c76" (UID: "f8da5d76-1168-40e9-97ae-b320a9b84c76"). InnerVolumeSpecName "kube-api-access-2866h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:01.753691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.753660    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-xks-ca" pod "f8da5d76-1168-40e9-97ae-b320a9b84c76" (UID: "f8da5d76-1168-40e9-97ae-b320a9b84c76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:38:01.854632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.854598    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2866h\" (UniqueName: \"kubernetes.io/projected/f8da5d76-1168-40e9-97ae-b320a9b84c76-kube-api-access-2866h\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:01.854632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.854627    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8da5d76-1168-40e9-97ae-b320a9b84c76-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:01.924388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.924261    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"
Oct 29 22:38:01.924388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.924318    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72" event={"ID":"f8da5d76-1168-40e9-97ae-b320a9b84c76","Type":"ContainerDied","Data":"afab7e868010125b47870c704a7e503494d75f4fb2f323b9d4a544c050b101f0"}
Oct 29 22:38:01.924620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.924414    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afab7e868010125b47870c704a7e503494d75f4fb2f323b9d4a544c050b101f0"
Oct 29 22:38:01.926320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.926293    4860 generic.go:356] "Generic (PLEG): container finished" podID="d1818f73-498f-47fd-afab-467d76cf7209" containerID="f11630547b811653b0184291134d22b10542abbc3713dfc44d12d654e16e0253" exitCode=0
Oct 29 22:38:01.926557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:01.926344    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb" event={"ID":"d1818f73-498f-47fd-afab-467d76cf7209","Type":"ContainerDied","Data":"f11630547b811653b0184291134d22b10542abbc3713dfc44d12d654e16e0253"}
Oct 29 22:38:02.628350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:02.628037    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:02.728663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:02.728628    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:38:02.728839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:02.728690    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 29 22:38:02.733176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:02.733153    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:38:02.733581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:02.733556    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:38:02.734068 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:02.734050    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:38:03.186981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.186956    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:38:03.303926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.303839    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-xks-ca\") pod \"d1818f73-498f-47fd-afab-467d76cf7209\" (UID: \"d1818f73-498f-47fd-afab-467d76cf7209\") "
Oct 29 22:38:03.304082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.304031    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1818f73-498f-47fd-afab-467d76cf7209-kube-api-access-dkr8n\" (UniqueName: \"kubernetes.io/projected/d1818f73-498f-47fd-afab-467d76cf7209-kube-api-access-dkr8n\") pod \"d1818f73-498f-47fd-afab-467d76cf7209\" (UID: \"d1818f73-498f-47fd-afab-467d76cf7209\") "
Oct 29 22:38:03.304082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.304066    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-openshift-ca\") pod \"d1818f73-498f-47fd-afab-467d76cf7209\" (UID: \"d1818f73-498f-47fd-afab-467d76cf7209\") "
Oct 29 22:38:03.304514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.304478    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-openshift-ca" pod "d1818f73-498f-47fd-afab-467d76cf7209" (UID: "d1818f73-498f-47fd-afab-467d76cf7209"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:03.305008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.304987    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:03.306783 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.306752    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-xks-ca" pod "d1818f73-498f-47fd-afab-467d76cf7209" (UID: "d1818f73-498f-47fd-afab-467d76cf7209"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:38:03.306783 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.306773    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1818f73-498f-47fd-afab-467d76cf7209-kube-api-access-dkr8n" pod "d1818f73-498f-47fd-afab-467d76cf7209" (UID: "d1818f73-498f-47fd-afab-467d76cf7209"). InnerVolumeSpecName "kube-api-access-dkr8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:03.406878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.406850    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1818f73-498f-47fd-afab-467d76cf7209-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:03.406878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.406878    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkr8n\" (UniqueName: \"kubernetes.io/projected/d1818f73-498f-47fd-afab-467d76cf7209-kube-api-access-dkr8n\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:03.773696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.773122    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:38:03.774696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.774621    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:38:03.776232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.776204    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:38:03.988126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.988093    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"
Oct 29 22:38:03.988126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.988108    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb" event={"ID":"d1818f73-498f-47fd-afab-467d76cf7209","Type":"ContainerDied","Data":"19ec72c74a06c422e4521b478aec01594350dd5786cd01bbd7f2ae889087ba92"}
Oct 29 22:38:03.988371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:03.988191    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19ec72c74a06c422e4521b478aec01594350dd5786cd01bbd7f2ae889087ba92"
Oct 29 22:38:04.546782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:04.546749    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:06.105779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:06.105751    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:06.310215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:06.310177    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:07.021589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:07.021560    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"]
Oct 29 22:38:07.021782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:07.021748    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s" podUID="9bb10043-0048-4595-bc98-b3333f888d5f" containerName="istio-proxy" containerID="cri-o://c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535" gracePeriod=30
Oct 29 22:38:07.035585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:07.035558    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"]
Oct 29 22:38:07.035805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:07.035782    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy" containerID="cri-o://b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480" gracePeriod=30
Oct 29 22:38:07.043822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:07.043538    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"]
Oct 29 22:38:07.043822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:07.043745    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w" podUID="1bb026b1-e8ce-461a-bcdc-a38955de1842" containerName="istio-proxy" containerID="cri-o://0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e" gracePeriod=30
Oct 29 22:38:07.060856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:07.060827    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"]
Oct 29 22:38:07.061062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:07.061037    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt" podUID="14cc6760-f3f7-45bd-9c31-655c607c4505" containerName="istio-proxy" containerID="cri-o://f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0" gracePeriod=30
Oct 29 22:38:07.469963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:07.469923    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:07.955960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:07.955917    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:08.095346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.090784    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:08.331816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.331429    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:08.351884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.351851    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"]
Oct 29 22:38:08.353614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.353588    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d1818f73-498f-47fd-afab-467d76cf7209" containerName="revoke-keys"
Oct 29 22:38:08.354493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.354474    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1818f73-498f-47fd-afab-467d76cf7209" containerName="revoke-keys"
Oct 29 22:38:08.354681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.354667    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f8da5d76-1168-40e9-97ae-b320a9b84c76" containerName="revoke-keys"
Oct 29 22:38:08.354779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.354767    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8da5d76-1168-40e9-97ae-b320a9b84c76" containerName="revoke-keys"
Oct 29 22:38:08.355228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.355210    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1818f73-498f-47fd-afab-467d76cf7209" containerName="revoke-keys"
Oct 29 22:38:08.355375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.355349    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8da5d76-1168-40e9-97ae-b320a9b84c76" containerName="revoke-keys"
Oct 29 22:38:08.389308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.386959    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"]
Oct 29 22:38:08.389308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.387012    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"]
Oct 29 22:38:08.389308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.387033    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1131a29d-f6gkg"]
Oct 29 22:38:08.389308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.387126    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:08.405501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.405461    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-aeda8c-serving-cert\""
Oct 29 22:38:08.405681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.405593    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dzwhk\""
Oct 29 22:38:08.419062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.419041    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:38:08.419168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.419054    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 22:38:08.419168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.419084    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:38:08.453290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.453261    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a154999-c757-467d-b8be-78e122dea6e6-openshift-service-ca\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:08.453455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.453405    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-metrics-certs\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:08.453508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.453481    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:08.453550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.453534    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llrll\" (UniqueName: \"kubernetes.io/projected/6a154999-c757-467d-b8be-78e122dea6e6-kube-api-access-llrll\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:08.556008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.555972    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a154999-c757-467d-b8be-78e122dea6e6-openshift-service-ca\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:08.556398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.556088    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-metrics-certs\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:08.556398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.556155    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:08.556398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.556217    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llrll\" (UniqueName: \"kubernetes.io/projected/6a154999-c757-467d-b8be-78e122dea6e6-kube-api-access-llrll\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:08.556681 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:08.556655    4860 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-serving-cert: secret "maas-api-e2e-api-b-aeda8c-serving-cert" not found
Oct 29 22:38:08.556801 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:08.556746    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls podName:6a154999-c757-467d-b8be-78e122dea6e6 nodeName:}" failed. No retries permitted until 2026-10-29 22:38:09.056724311 +0000 UTC m=+1431.081500428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls") pod "maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd" (UID: "6a154999-c757-467d-b8be-78e122dea6e6") : secret "maas-api-e2e-api-b-aeda8c-serving-cert" not found
Oct 29 22:38:08.557238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.557214    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a154999-c757-467d-b8be-78e122dea6e6-openshift-service-ca\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:08.559926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.559900    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-metrics-certs\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:08.578364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.578298    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llrll\" (UniqueName: \"kubernetes.io/projected/6a154999-c757-467d-b8be-78e122dea6e6-kube-api-access-llrll\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:08.656777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:08.656696    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:09.066485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:09.066450    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:09.066739 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:09.066623    4860 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-serving-cert: secret "maas-api-e2e-api-b-aeda8c-serving-cert" not found
Oct 29 22:38:09.066739 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:09.066707    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls podName:6a154999-c757-467d-b8be-78e122dea6e6 nodeName:}" failed. No retries permitted until 2026-10-29 22:38:10.066683447 +0000 UTC m=+1432.091459557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls") pod "maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd" (UID: "6a154999-c757-467d-b8be-78e122dea6e6") : secret "maas-api-e2e-api-b-aeda8c-serving-cert" not found
Oct 29 22:38:09.482999 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:09.482957    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:10.116663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:10.116628    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06dd2821-5329-47e0-a877-6fd99dcc8b3e" path="/var/lib/kubelet/pods/06dd2821-5329-47e0-a877-6fd99dcc8b3e/volumes"
Oct 29 22:38:10.121621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:10.121593    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:10.124081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:10.124054    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls\") pod \"maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:10.144997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:10.144965    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:10.205725 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:10.205686    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:10.531853 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:38:10.531818    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a154999_c757_467d_b8be_78e122dea6e6.slice/crio-d0db2dc4d50c3503573560eb305b7847b7862c07fa585ab1482ddafcb6931be3 WatchSource:0}: Error finding container d0db2dc4d50c3503573560eb305b7847b7862c07fa585ab1482ddafcb6931be3: Status 404 returned error can't find the container with id d0db2dc4d50c3503573560eb305b7847b7862c07fa585ab1482ddafcb6931be3
Oct 29 22:38:10.536629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:10.536605    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"]
Oct 29 22:38:11.105844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:11.105808    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd" event={"ID":"6a154999-c757-467d-b8be-78e122dea6e6","Type":"ContainerStarted","Data":"d0db2dc4d50c3503573560eb305b7847b7862c07fa585ab1482ddafcb6931be3"}
Oct 29 22:38:11.852808 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:11.852775    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:11.983283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:11.983245    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:12.037923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.036727    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"]
Oct 29 22:38:12.048553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.048520    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555e24ad-n5cfz"]
Oct 29 22:38:12.127170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.127139    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7bb2813-2dbf-4245-92cd-a1278a0f48a7" path="/var/lib/kubelet/pods/c7bb2813-2dbf-4245-92cd-a1278a0f48a7/volumes"
Oct 29 22:38:12.412940 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.412909    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:38:12.427953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.427930    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.472756    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-certs\") pod \"1bb026b1-e8ce-461a-bcdc-a38955de1842\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.472809    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-data\") pod \"9bb10043-0048-4595-bc98-b3333f888d5f\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.472902    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb026b1-e8ce-461a-bcdc-a38955de1842-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb026b1-e8ce-461a-bcdc-a38955de1842-istiod-ca-cert\") pod \"1bb026b1-e8ce-461a-bcdc-a38955de1842\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473000    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-envoy\") pod \"1bb026b1-e8ce-461a-bcdc-a38955de1842\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473033    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-socket\") pod \"9bb10043-0048-4595-bc98-b3333f888d5f\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473067    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-certs\") pod \"9bb10043-0048-4595-bc98-b3333f888d5f\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473120    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-socket\") pod \"1bb026b1-e8ce-461a-bcdc-a38955de1842\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473165    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-kube-api-access-mch5r\" (UniqueName: \"kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-kube-api-access-mch5r\") pod \"9bb10043-0048-4595-bc98-b3333f888d5f\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473217    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bb10043-0048-4595-bc98-b3333f888d5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb10043-0048-4595-bc98-b3333f888d5f-istio-podinfo\") pod \"9bb10043-0048-4595-bc98-b3333f888d5f\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473260    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-istio-token\" (UniqueName: \"kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-istio-token\") pod \"9bb10043-0048-4595-bc98-b3333f888d5f\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473385    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-credential-socket\") pod \"1bb026b1-e8ce-461a-bcdc-a38955de1842\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473416    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-data\") pod \"1bb026b1-e8ce-461a-bcdc-a38955de1842\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473454    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb10043-0048-4595-bc98-b3333f888d5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb10043-0048-4595-bc98-b3333f888d5f-istiod-ca-cert\") pod \"9bb10043-0048-4595-bc98-b3333f888d5f\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473592    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-credential-socket\") pod \"9bb10043-0048-4595-bc98-b3333f888d5f\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473766    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-envoy\") pod \"9bb10043-0048-4595-bc98-b3333f888d5f\" (UID: \"9bb10043-0048-4595-bc98-b3333f888d5f\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473827    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-podinfo\") pod \"1bb026b1-e8ce-461a-bcdc-a38955de1842\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473870    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-kube-api-access-fvszx\" (UniqueName: \"kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-kube-api-access-fvszx\") pod \"1bb026b1-e8ce-461a-bcdc-a38955de1842\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.473917    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-token\" (UniqueName: \"kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-token\") pod \"1bb026b1-e8ce-461a-bcdc-a38955de1842\" (UID: \"1bb026b1-e8ce-461a-bcdc-a38955de1842\") "
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.475545    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-certs" pod "9bb10043-0048-4595-bc98-b3333f888d5f" (UID: "9bb10043-0048-4595-bc98-b3333f888d5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.475739    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-socket" pod "9bb10043-0048-4595-bc98-b3333f888d5f" (UID: "9bb10043-0048-4595-bc98-b3333f888d5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.475750    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-socket" pod "1bb026b1-e8ce-461a-bcdc-a38955de1842" (UID: "1bb026b1-e8ce-461a-bcdc-a38955de1842"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.476071    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-certs" pod "1bb026b1-e8ce-461a-bcdc-a38955de1842" (UID: "1bb026b1-e8ce-461a-bcdc-a38955de1842"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.476090    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb026b1-e8ce-461a-bcdc-a38955de1842-istiod-ca-cert" pod "1bb026b1-e8ce-461a-bcdc-a38955de1842" (UID: "1bb026b1-e8ce-461a-bcdc-a38955de1842"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.476371    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-credential-socket" pod "1bb026b1-e8ce-461a-bcdc-a38955de1842" (UID: "1bb026b1-e8ce-461a-bcdc-a38955de1842"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.476416    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-data" pod "9bb10043-0048-4595-bc98-b3333f888d5f" (UID: "9bb10043-0048-4595-bc98-b3333f888d5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.476542    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb10043-0048-4595-bc98-b3333f888d5f-istiod-ca-cert" pod "9bb10043-0048-4595-bc98-b3333f888d5f" (UID: "9bb10043-0048-4595-bc98-b3333f888d5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.477008    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-data" pod "1bb026b1-e8ce-461a-bcdc-a38955de1842" (UID: "1bb026b1-e8ce-461a-bcdc-a38955de1842"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.479187    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-credential-socket" pod "9bb10043-0048-4595-bc98-b3333f888d5f" (UID: "9bb10043-0048-4595-bc98-b3333f888d5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.479362    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-kube-api-access-mch5r" pod "9bb10043-0048-4595-bc98-b3333f888d5f" (UID: "9bb10043-0048-4595-bc98-b3333f888d5f"). InnerVolumeSpecName "kube-api-access-mch5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:12.481380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.479770    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-envoy" pod "9bb10043-0048-4595-bc98-b3333f888d5f" (UID: "9bb10043-0048-4595-bc98-b3333f888d5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.498955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.498928    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-istio-token" pod "9bb10043-0048-4595-bc98-b3333f888d5f" (UID: "9bb10043-0048-4595-bc98-b3333f888d5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:12.499116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.499091    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-envoy" pod "1bb026b1-e8ce-461a-bcdc-a38955de1842" (UID: "1bb026b1-e8ce-461a-bcdc-a38955de1842"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.499513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.499476    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bb10043-0048-4595-bc98-b3333f888d5f-istio-podinfo" pod "9bb10043-0048-4595-bc98-b3333f888d5f" (UID: "9bb10043-0048-4595-bc98-b3333f888d5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:12.499632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.499593    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-token" pod "1bb026b1-e8ce-461a-bcdc-a38955de1842" (UID: "1bb026b1-e8ce-461a-bcdc-a38955de1842"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:12.499697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.499645    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-podinfo" pod "1bb026b1-e8ce-461a-bcdc-a38955de1842" (UID: "1bb026b1-e8ce-461a-bcdc-a38955de1842"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:12.502709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.502681    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-kube-api-access-fvszx" pod "1bb026b1-e8ce-461a-bcdc-a38955de1842" (UID: "1bb026b1-e8ce-461a-bcdc-a38955de1842"). InnerVolumeSpecName "kube-api-access-fvszx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:12.576865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.576829    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.576865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.576867    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.576881    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.576897    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvszx\" (UniqueName: \"kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-kube-api-access-fvszx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.576911    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.576924    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.576936    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.576949    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bb026b1-e8ce-461a-bcdc-a38955de1842-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.576962    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.576976    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.576988    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb10043-0048-4595-bc98-b3333f888d5f-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.577003    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.577015    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mch5r\" (UniqueName: \"kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-kube-api-access-mch5r\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.577030    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb10043-0048-4595-bc98-b3333f888d5f-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.577042    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb10043-0048-4595-bc98-b3333f888d5f-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.577054    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.577065    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bb026b1-e8ce-461a-bcdc-a38955de1842-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.577815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.577077    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb10043-0048-4595-bc98-b3333f888d5f-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.698389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.697111    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:12.760296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.759817    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:38:12.828829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.828800    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:12.833228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.833206    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:38:12.882947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.882891    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-podinfo\") pod \"14cc6760-f3f7-45bd-9c31-655c607c4505\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") "
Oct 29 22:38:12.883425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.882949    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-credential-socket\") pod \"14cc6760-f3f7-45bd-9c31-655c607c4505\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") "
Oct 29 22:38:12.883425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.882979    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-kube-api-access-vmjxt\" (UniqueName: \"kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-kube-api-access-vmjxt\") pod \"4e41504b-834f-4cc2-89f9-ed6b7338f644\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") "
Oct 29 22:38:12.883425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883025    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-token\" (UniqueName: \"kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-token\") pod \"14cc6760-f3f7-45bd-9c31-655c607c4505\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") "
Oct 29 22:38:12.883425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883076    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-envoy\") pod \"14cc6760-f3f7-45bd-9c31-655c607c4505\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") "
Oct 29 22:38:12.883425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883104    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cc6760-f3f7-45bd-9c31-655c607c4505-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cc6760-f3f7-45bd-9c31-655c607c4505-istiod-ca-cert\") pod \"14cc6760-f3f7-45bd-9c31-655c607c4505\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") "
Oct 29 22:38:12.883425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883179    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-envoy\") pod \"4e41504b-834f-4cc2-89f9-ed6b7338f644\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") "
Oct 29 22:38:12.883425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883248    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e41504b-834f-4cc2-89f9-ed6b7338f644-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e41504b-834f-4cc2-89f9-ed6b7338f644-istiod-ca-cert\") pod \"4e41504b-834f-4cc2-89f9-ed6b7338f644\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") "
Oct 29 22:38:12.883425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883291    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-data\") pod \"4e41504b-834f-4cc2-89f9-ed6b7338f644\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") "
Oct 29 22:38:12.883425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883365    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-credential-socket\") pod \"4e41504b-834f-4cc2-89f9-ed6b7338f644\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") "
Oct 29 22:38:12.883425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883398    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-certs\") pod \"4e41504b-834f-4cc2-89f9-ed6b7338f644\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") "
Oct 29 22:38:12.883934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883459    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-token\" (UniqueName: \"kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-token\") pod \"4e41504b-834f-4cc2-89f9-ed6b7338f644\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") "
Oct 29 22:38:12.883934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883583    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-socket\") pod \"14cc6760-f3f7-45bd-9c31-655c607c4505\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") "
Oct 29 22:38:12.883934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883696    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-podinfo\") pod \"4e41504b-834f-4cc2-89f9-ed6b7338f644\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") "
Oct 29 22:38:12.883934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883790    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-socket\") pod \"4e41504b-834f-4cc2-89f9-ed6b7338f644\" (UID: \"4e41504b-834f-4cc2-89f9-ed6b7338f644\") "
Oct 29 22:38:12.884133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883939    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-certs\") pod \"14cc6760-f3f7-45bd-9c31-655c607c4505\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") "
Oct 29 22:38:12.884133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.883997    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-kube-api-access-8vrrb\" (UniqueName: \"kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-kube-api-access-8vrrb\") pod \"14cc6760-f3f7-45bd-9c31-655c607c4505\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") "
Oct 29 22:38:12.884133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.884063    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-data\") pod \"14cc6760-f3f7-45bd-9c31-655c607c4505\" (UID: \"14cc6760-f3f7-45bd-9c31-655c607c4505\") "
Oct 29 22:38:12.886533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.884823    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-data" pod "4e41504b-834f-4cc2-89f9-ed6b7338f644" (UID: "4e41504b-834f-4cc2-89f9-ed6b7338f644"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.886533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.885090    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.886533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.885584    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-data" pod "14cc6760-f3f7-45bd-9c31-655c607c4505" (UID: "14cc6760-f3f7-45bd-9c31-655c607c4505"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.886533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.885760    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-credential-socket" pod "4e41504b-834f-4cc2-89f9-ed6b7338f644" (UID: "4e41504b-834f-4cc2-89f9-ed6b7338f644"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.886533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.885910    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e41504b-834f-4cc2-89f9-ed6b7338f644-istiod-ca-cert" pod "4e41504b-834f-4cc2-89f9-ed6b7338f644" (UID: "4e41504b-834f-4cc2-89f9-ed6b7338f644"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:12.886533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.886003    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cc6760-f3f7-45bd-9c31-655c607c4505-istiod-ca-cert" pod "14cc6760-f3f7-45bd-9c31-655c607c4505" (UID: "14cc6760-f3f7-45bd-9c31-655c607c4505"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:12.886533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.886051    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-credential-socket" pod "14cc6760-f3f7-45bd-9c31-655c607c4505" (UID: "14cc6760-f3f7-45bd-9c31-655c607c4505"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.886533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.886055    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-certs" pod "4e41504b-834f-4cc2-89f9-ed6b7338f644" (UID: "4e41504b-834f-4cc2-89f9-ed6b7338f644"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.886533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.886260    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-certs" pod "14cc6760-f3f7-45bd-9c31-655c607c4505" (UID: "14cc6760-f3f7-45bd-9c31-655c607c4505"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.886533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.886281    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-socket" pod "4e41504b-834f-4cc2-89f9-ed6b7338f644" (UID: "4e41504b-834f-4cc2-89f9-ed6b7338f644"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.887253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.886860    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-socket" pod "14cc6760-f3f7-45bd-9c31-655c607c4505" (UID: "14cc6760-f3f7-45bd-9c31-655c607c4505"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.892884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.892843    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-podinfo" pod "4e41504b-834f-4cc2-89f9-ed6b7338f644" (UID: "4e41504b-834f-4cc2-89f9-ed6b7338f644"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:12.892884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.892852    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-envoy" pod "14cc6760-f3f7-45bd-9c31-655c607c4505" (UID: "14cc6760-f3f7-45bd-9c31-655c607c4505"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.893024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.892908    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-kube-api-access-vmjxt" pod "4e41504b-834f-4cc2-89f9-ed6b7338f644" (UID: "4e41504b-834f-4cc2-89f9-ed6b7338f644"). InnerVolumeSpecName "kube-api-access-vmjxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:12.893427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.893401    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-kube-api-access-8vrrb" pod "14cc6760-f3f7-45bd-9c31-655c607c4505" (UID: "14cc6760-f3f7-45bd-9c31-655c607c4505"). InnerVolumeSpecName "kube-api-access-8vrrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:12.893601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.893550    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-token" pod "14cc6760-f3f7-45bd-9c31-655c607c4505" (UID: "14cc6760-f3f7-45bd-9c31-655c607c4505"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:12.893971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.893943    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-token" pod "4e41504b-834f-4cc2-89f9-ed6b7338f644" (UID: "4e41504b-834f-4cc2-89f9-ed6b7338f644"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:12.894395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.894371    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-podinfo" pod "14cc6760-f3f7-45bd-9c31-655c607c4505" (UID: "14cc6760-f3f7-45bd-9c31-655c607c4505"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:12.895577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.895552    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-envoy" pod "4e41504b-834f-4cc2-89f9-ed6b7338f644" (UID: "4e41504b-834f-4cc2-89f9-ed6b7338f644"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987740    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987773    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987787    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987803    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vrrb\" (UniqueName: \"kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-kube-api-access-8vrrb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987816    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987829    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987841    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987853    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjxt\" (UniqueName: \"kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-kube-api-access-vmjxt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987865    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987879    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987893    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cc6760-f3f7-45bd-9c31-655c607c4505-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987904    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987916    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e41504b-834f-4cc2-89f9-ed6b7338f644-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987927    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987941    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e41504b-834f-4cc2-89f9-ed6b7338f644-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987954    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e41504b-834f-4cc2-89f9-ed6b7338f644-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:12.987987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:12.987965    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cc6760-f3f7-45bd-9c31-655c607c4505-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:13.148318 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.148265    4860 generic.go:356] "Generic (PLEG): container finished" podID="14cc6760-f3f7-45bd-9c31-655c607c4505" containerID="f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0" exitCode=0
Oct 29 22:38:13.148546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.148276    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt" event={"ID":"14cc6760-f3f7-45bd-9c31-655c607c4505","Type":"ContainerDied","Data":"f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0"}
Oct 29 22:38:13.148546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.148381    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"
Oct 29 22:38:13.148546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.148527    4860 scope.go:122] "RemoveContainer" containerID="f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0"
Oct 29 22:38:13.148820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.148763    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt" event={"ID":"14cc6760-f3f7-45bd-9c31-655c607c4505","Type":"ContainerDied","Data":"0a7de8e601e4ea4adbba2881d0551b15ed14096b1577f90128a20eaba87f8faf"}
Oct 29 22:38:13.151787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.151761    4860 generic.go:356] "Generic (PLEG): container finished" podID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerID="b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480" exitCode=0
Oct 29 22:38:13.151902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.151835    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"
Oct 29 22:38:13.151902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.151836    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" event={"ID":"4e41504b-834f-4cc2-89f9-ed6b7338f644","Type":"ContainerDied","Data":"b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480"}
Oct 29 22:38:13.151902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.151890    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc" event={"ID":"4e41504b-834f-4cc2-89f9-ed6b7338f644","Type":"ContainerDied","Data":"1888c197f2222c17ec528e1324609074d8d25e977de7db7886c354c25dfbe8ac"}
Oct 29 22:38:13.156378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.156191    4860 generic.go:356] "Generic (PLEG): container finished" podID="1bb026b1-e8ce-461a-bcdc-a38955de1842" containerID="0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e" exitCode=0
Oct 29 22:38:13.156378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.156280    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"
Oct 29 22:38:13.156550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.156452    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w" event={"ID":"1bb026b1-e8ce-461a-bcdc-a38955de1842","Type":"ContainerDied","Data":"0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e"}
Oct 29 22:38:13.156550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.156493    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w" event={"ID":"1bb026b1-e8ce-461a-bcdc-a38955de1842","Type":"ContainerDied","Data":"f46bfaa13c9be86bf3a10f2fc6ba6a59afc9a93a3decf49dc26f16b3eb6b0e44"}
Oct 29 22:38:13.158566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.158542    4860 generic.go:356] "Generic (PLEG): container finished" podID="9bb10043-0048-4595-bc98-b3333f888d5f" containerID="c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535" exitCode=0
Oct 29 22:38:13.158670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.158598    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s" event={"ID":"9bb10043-0048-4595-bc98-b3333f888d5f","Type":"ContainerDied","Data":"c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535"}
Oct 29 22:38:13.158670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.158633    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s" event={"ID":"9bb10043-0048-4595-bc98-b3333f888d5f","Type":"ContainerDied","Data":"89d723ad2769e8fa2acef75a88fc794d01619975b1071c1dc55994ac2bf48c8a"}
Oct 29 22:38:13.158777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.158716    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"
Oct 29 22:38:13.167198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.167021    4860 scope.go:122] "RemoveContainer" containerID="f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0"
Oct 29 22:38:13.167432 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:13.167319    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0\": container with ID starting with f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0 not found: ID does not exist" containerID="f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0"
Oct 29 22:38:13.167432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.167397    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0"} err="failed to get container status \"f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0\": rpc error: code = NotFound desc = could not find container \"f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0\": container with ID starting with f34d74c85f9a8753acfbf387606d3b21ceeca8b5c3dc3ac824af7fcd3681cea0 not found: ID does not exist"
Oct 29 22:38:13.167432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.167419    4860 scope.go:122] "RemoveContainer" containerID="b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480"
Oct 29 22:38:13.185073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.185051    4860 scope.go:122] "RemoveContainer" containerID="b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480"
Oct 29 22:38:13.185429 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:13.185400    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480\": container with ID starting with b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480 not found: ID does not exist" containerID="b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480"
Oct 29 22:38:13.185519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.185436    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480"} err="failed to get container status \"b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480\": rpc error: code = NotFound desc = could not find container \"b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480\": container with ID starting with b56c98209a0aa0f0aa7955d9b3a366e5601bdd9c4db203660168ce26f2459480 not found: ID does not exist"
Oct 29 22:38:13.185519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.185458    4860 scope.go:122] "RemoveContainer" containerID="0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e"
Oct 29 22:38:13.198721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.198637    4860 scope.go:122] "RemoveContainer" containerID="0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e"
Oct 29 22:38:13.199140 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:13.199012    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e\": container with ID starting with 0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e not found: ID does not exist" containerID="0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e"
Oct 29 22:38:13.199140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.199044    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e"} err="failed to get container status \"0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e\": rpc error: code = NotFound desc = could not find container \"0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e\": container with ID starting with 0fe96aaadc6b9f4fb96c03f9854caa4bf59c982c6bc767ae662b38a7ec96401e not found: ID does not exist"
Oct 29 22:38:13.199140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.199061    4860 scope.go:122] "RemoveContainer" containerID="c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535"
Oct 29 22:38:13.212067 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.212036    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"]
Oct 29 22:38:13.212754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.212734    4860 scope.go:122] "RemoveContainer" containerID="c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535"
Oct 29 22:38:13.213128 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:13.213109    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535\": container with ID starting with c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535 not found: ID does not exist" containerID="c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535"
Oct 29 22:38:13.213203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.213132    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535"} err="failed to get container status \"c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535\": rpc error: code = NotFound desc = could not find container \"c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535\": container with ID starting with c919209fae459a820b41ec92f01ab1ef83f12155f43e930a26f3b5cfde2b4535 not found: ID does not exist"
Oct 29 22:38:13.223945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.223925    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-jjlwt"]
Oct 29 22:38:13.308533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.308456    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"]
Oct 29 22:38:13.314542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.314511    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9hfvc"]
Oct 29 22:38:13.354840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.354811    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"]
Oct 29 22:38:13.361708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.361645    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-vt95w"]
Oct 29 22:38:13.397736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.397697    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"]
Oct 29 22:38:13.404634 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.404606    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2ba10a-openshift-default-6d84cb5d99-9rm2s"]
Oct 29 22:38:13.668336 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:13.668292    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:14.114886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:14.114845    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14cc6760-f3f7-45bd-9c31-655c607c4505" path="/var/lib/kubelet/pods/14cc6760-f3f7-45bd-9c31-655c607c4505/volumes"
Oct 29 22:38:14.115482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:14.115465    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb026b1-e8ce-461a-bcdc-a38955de1842" path="/var/lib/kubelet/pods/1bb026b1-e8ce-461a-bcdc-a38955de1842/volumes"
Oct 29 22:38:14.115993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:14.115979    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" path="/var/lib/kubelet/pods/4e41504b-834f-4cc2-89f9-ed6b7338f644/volumes"
Oct 29 22:38:14.116848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:14.116831    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb10043-0048-4595-bc98-b3333f888d5f" path="/var/lib/kubelet/pods/9bb10043-0048-4595-bc98-b3333f888d5f/volumes"
Oct 29 22:38:14.173972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:14.173935    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd" event={"ID":"6a154999-c757-467d-b8be-78e122dea6e6","Type":"ContainerStarted","Data":"b1f7e9c88dde03684d6d703dae779a1518e56ed22302cab59aefd2515d7c880e"}
Oct 29 22:38:14.174154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:14.174062    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:14.220712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:14.220653    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd" podStartSLOduration=3.25379345 podStartE2EDuration="6.220633491s" totalImagesPullingTime="2.966840041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:14.219671606 +0000 UTC m=+1436.244447740" watchObservedRunningTime="2026-10-29 22:38:14.220633491 +0000 UTC m=+1436.245409626"
Oct 29 22:38:15.082690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:15.082655    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:15.312875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:15.312799    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:15.437356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:15.435465    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"]
Oct 29 22:38:15.437356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:15.435662    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerName="istio-proxy" containerID="cri-o://84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164" gracePeriod=30
Oct 29 22:38:15.442610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:15.440910    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"]
Oct 29 22:38:15.442610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:15.441119    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" podUID="cd9fabed-9589-46a6-843a-6fd99d1278ff" containerName="istio-proxy" containerID="cri-o://8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366" gracePeriod=30
Oct 29 22:38:15.467078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:15.467044    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:15.674441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:15.671442    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:16.421413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:16.421378    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:17.004145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:17.004089    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:17.362692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:17.362581    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:17.479374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:17.479311    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:17.736218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:17.736185    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 29 22:38:17.736404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:17.736250    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" podUID="cd9fabed-9589-46a6-843a-6fd99d1278ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:38:18.059230 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:18.059150    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:38:18.777687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:18.777649    4860 patch_prober.go:29] interesting pod/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:38:18.778238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:18.777712    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:38:19.061786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.061689    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:19.065251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.065226    4860 scope.go:122] "RemoveContainer" containerID="a3cd4ca44e1fd6f1d334a9a4ce03de76ee9b09fc9e2081de27a392c94521f31e"
Oct 29 22:38:19.127628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.127588    4860 scope.go:122] "RemoveContainer" containerID="4814adf441cb06d860e3538985f1d8969a8ab7229f29e5a0f0ff52c2546a9779"
Oct 29 22:38:19.197383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.197304    4860 scope.go:122] "RemoveContainer" containerID="99cda612383bf7dbb089fde1a73245174e6143f8b09d91f55577eb377df999c6"
Oct 29 22:38:19.214156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.214124    4860 scope.go:122] "RemoveContainer" containerID="f5821400627e979e250cfb51572266749dcc1f08faefdc06f1bef07ffdfd3e28"
Oct 29 22:38:19.226465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.226437    4860 scope.go:122] "RemoveContainer" containerID="4907a7d2a599eedf383905580508f4bdc26cfcc925fc813d99cb9e145147a453"
Oct 29 22:38:19.239423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.239389    4860 scope.go:122] "RemoveContainer" containerID="454c1029a72150d44be1f6142b8997caa5bd4df7c7dad3208c23fdb2adc9f1c2"
Oct 29 22:38:19.253282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.253256    4860 scope.go:122] "RemoveContainer" containerID="5fb3e05a0e073d06611cc0ca02fd64e95a6299d54dee794d971eeff6fe094b93"
Oct 29 22:38:19.265376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.265349    4860 scope.go:122] "RemoveContainer" containerID="8c38771ebf9d3b36cf97cf1576940d2f9d4c6fd3198430463f2e14b80145cc62"
Oct 29 22:38:19.278477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.278458    4860 scope.go:122] "RemoveContainer" containerID="1d102096c87c5d0f4d3f29b882f82a8b19200fd73f5c5e6c521e798d11cb834b"
Oct 29 22:38:19.289706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.289689    4860 scope.go:122] "RemoveContainer" containerID="4d15d62203cca57986eaf806f6f801c26019c3d352241cdeda3f79ae76257e99"
Oct 29 22:38:19.301885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.301859    4860 scope.go:122] "RemoveContainer" containerID="da10ffcd693f5d618bf1ef3197d2cc4d960691428b56a2295ff8dc9b6aa07845"
Oct 29 22:38:19.312439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.312414    4860 scope.go:122] "RemoveContainer" containerID="a5454b254abba345333daafe77013300deadd340da40a1dd54e288cf7243f24a"
Oct 29 22:38:19.323726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.323707    4860 scope.go:122] "RemoveContainer" containerID="83ad9f7b7e9644e62e03921c880f5e081d1a1c33fd2c0119eaf2974f90e0b4ab"
Oct 29 22:38:19.334484 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:19.334466    4860 scope.go:122] "RemoveContainer" containerID="70c5020fb95e96050b2248114548f7ed7be7c5e04f2325881bca31950751df66"
Oct 29 22:38:20.197125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.197099    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:38:20.793754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.791829    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:38:20.799101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.799070    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:38:20.833081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833042    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-podinfo\") pod \"05b8f3bc-6cc7-44e3-b406-d278376a586a\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") "
Oct 29 22:38:20.833281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833088    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-data\") pod \"cd9fabed-9589-46a6-843a-6fd99d1278ff\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") "
Oct 29 22:38:20.833281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833135    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-kube-api-access-xkxpr\" (UniqueName: \"kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-kube-api-access-xkxpr\") pod \"05b8f3bc-6cc7-44e3-b406-d278376a586a\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") "
Oct 29 22:38:20.833281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833273    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-token\" (UniqueName: \"kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-token\") pod \"cd9fabed-9589-46a6-843a-6fd99d1278ff\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") "
Oct 29 22:38:20.833500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833340    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-socket\") pod \"cd9fabed-9589-46a6-843a-6fd99d1278ff\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") "
Oct 29 22:38:20.833500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833391    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-token\" (UniqueName: \"kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-token\") pod \"05b8f3bc-6cc7-44e3-b406-d278376a586a\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") "
Oct 29 22:38:20.833500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833483    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-certs\") pod \"cd9fabed-9589-46a6-843a-6fd99d1278ff\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") "
Oct 29 22:38:20.833670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833516    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-socket\") pod \"05b8f3bc-6cc7-44e3-b406-d278376a586a\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") "
Oct 29 22:38:20.833670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833589    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-podinfo\") pod \"cd9fabed-9589-46a6-843a-6fd99d1278ff\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") "
Oct 29 22:38:20.833772 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833694    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-kube-api-access-dw84v\" (UniqueName: \"kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-kube-api-access-dw84v\") pod \"cd9fabed-9589-46a6-843a-6fd99d1278ff\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") "
Oct 29 22:38:20.833772 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833740    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-envoy\") pod \"05b8f3bc-6cc7-44e3-b406-d278376a586a\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") "
Oct 29 22:38:20.833878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833789    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9fabed-9589-46a6-843a-6fd99d1278ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9fabed-9589-46a6-843a-6fd99d1278ff-istiod-ca-cert\") pod \"cd9fabed-9589-46a6-843a-6fd99d1278ff\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") "
Oct 29 22:38:20.833936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833903    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-credential-socket\") pod \"cd9fabed-9589-46a6-843a-6fd99d1278ff\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") "
Oct 29 22:38:20.833988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.833961    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-credential-socket\") pod \"05b8f3bc-6cc7-44e3-b406-d278376a586a\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") "
Oct 29 22:38:20.834118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.834097    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-certs\") pod \"05b8f3bc-6cc7-44e3-b406-d278376a586a\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") "
Oct 29 22:38:20.834196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.834150    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-data\") pod \"05b8f3bc-6cc7-44e3-b406-d278376a586a\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") "
Oct 29 22:38:20.834196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.834185    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-envoy\") pod \"cd9fabed-9589-46a6-843a-6fd99d1278ff\" (UID: \"cd9fabed-9589-46a6-843a-6fd99d1278ff\") "
Oct 29 22:38:20.834301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.834226    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05b8f3bc-6cc7-44e3-b406-d278376a586a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b8f3bc-6cc7-44e3-b406-d278376a586a-istiod-ca-cert\") pod \"05b8f3bc-6cc7-44e3-b406-d278376a586a\" (UID: \"05b8f3bc-6cc7-44e3-b406-d278376a586a\") "
Oct 29 22:38:20.834785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.834426    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-socket" pod "cd9fabed-9589-46a6-843a-6fd99d1278ff" (UID: "cd9fabed-9589-46a6-843a-6fd99d1278ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.834785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.834656    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-socket" pod "05b8f3bc-6cc7-44e3-b406-d278376a586a" (UID: "05b8f3bc-6cc7-44e3-b406-d278376a586a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.834998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.834823    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-data" pod "cd9fabed-9589-46a6-843a-6fd99d1278ff" (UID: "cd9fabed-9589-46a6-843a-6fd99d1278ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.834998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.834888    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-certs" pod "cd9fabed-9589-46a6-843a-6fd99d1278ff" (UID: "cd9fabed-9589-46a6-843a-6fd99d1278ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.834998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.834956    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b8f3bc-6cc7-44e3-b406-d278376a586a-istiod-ca-cert" pod "05b8f3bc-6cc7-44e3-b406-d278376a586a" (UID: "05b8f3bc-6cc7-44e3-b406-d278376a586a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:20.835200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.835131    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-credential-socket" pod "05b8f3bc-6cc7-44e3-b406-d278376a586a" (UID: "05b8f3bc-6cc7-44e3-b406-d278376a586a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.835337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.835300    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-certs" pod "05b8f3bc-6cc7-44e3-b406-d278376a586a" (UID: "05b8f3bc-6cc7-44e3-b406-d278376a586a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.835450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.835429    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-data" pod "05b8f3bc-6cc7-44e3-b406-d278376a586a" (UID: "05b8f3bc-6cc7-44e3-b406-d278376a586a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.836054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.836027    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-credential-socket" pod "cd9fabed-9589-46a6-843a-6fd99d1278ff" (UID: "cd9fabed-9589-46a6-843a-6fd99d1278ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.836214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.836192    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9fabed-9589-46a6-843a-6fd99d1278ff-istiod-ca-cert" pod "cd9fabed-9589-46a6-843a-6fd99d1278ff" (UID: "cd9fabed-9589-46a6-843a-6fd99d1278ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:20.836456 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.836434    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.836539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.836462    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.836539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.836479    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.836539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.836492    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd9fabed-9589-46a6-843a-6fd99d1278ff-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.836539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.836521    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.836539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.836535    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.836809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.836550    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.836809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.836563    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.836809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.836576    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b8f3bc-6cc7-44e3-b406-d278376a586a-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.836809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.836588    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.841455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.841433    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-envoy" pod "05b8f3bc-6cc7-44e3-b406-d278376a586a" (UID: "05b8f3bc-6cc7-44e3-b406-d278376a586a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.842867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.842713    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-podinfo" pod "cd9fabed-9589-46a6-843a-6fd99d1278ff" (UID: "cd9fabed-9589-46a6-843a-6fd99d1278ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:20.842867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.842766    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-kube-api-access-dw84v" pod "cd9fabed-9589-46a6-843a-6fd99d1278ff" (UID: "cd9fabed-9589-46a6-843a-6fd99d1278ff"). InnerVolumeSpecName "kube-api-access-dw84v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:20.842867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.842826    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-kube-api-access-xkxpr" pod "05b8f3bc-6cc7-44e3-b406-d278376a586a" (UID: "05b8f3bc-6cc7-44e3-b406-d278376a586a"). InnerVolumeSpecName "kube-api-access-xkxpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:20.843125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.842932    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-token" pod "05b8f3bc-6cc7-44e3-b406-d278376a586a" (UID: "05b8f3bc-6cc7-44e3-b406-d278376a586a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:20.843554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.843482    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-podinfo" pod "05b8f3bc-6cc7-44e3-b406-d278376a586a" (UID: "05b8f3bc-6cc7-44e3-b406-d278376a586a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:20.843554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.843517    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-token" pod "cd9fabed-9589-46a6-843a-6fd99d1278ff" (UID: "cd9fabed-9589-46a6-843a-6fd99d1278ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:20.844622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.844591    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-envoy" pod "cd9fabed-9589-46a6-843a-6fd99d1278ff" (UID: "cd9fabed-9589-46a6-843a-6fd99d1278ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:20.942790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.942753    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw84v\" (UniqueName: \"kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-kube-api-access-dw84v\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.942790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.942790    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.943048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.942805    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.943048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.942819    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.943048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.942832    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkxpr\" (UniqueName: \"kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-kube-api-access-xkxpr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.943048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.942848    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.943048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.942861    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05b8f3bc-6cc7-44e3-b406-d278376a586a-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:20.943048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:20.942873    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd9fabed-9589-46a6-843a-6fd99d1278ff-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:21.017664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.016441    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:21.346840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.346754    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:21.396347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.394721    4860 generic.go:356] "Generic (PLEG): container finished" podID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerID="84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164" exitCode=0
Oct 29 22:38:21.396347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.395058    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" event={"ID":"05b8f3bc-6cc7-44e3-b406-d278376a586a","Type":"ContainerDied","Data":"84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164"}
Oct 29 22:38:21.396347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.395151    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd" event={"ID":"05b8f3bc-6cc7-44e3-b406-d278376a586a","Type":"ContainerDied","Data":"7e98b95f8290de4121b41fe0049635f488d6a0a6ca71b0b2635602516d44b2e8"}
Oct 29 22:38:21.396347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.395182    4860 scope.go:122] "RemoveContainer" containerID="84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164"
Oct 29 22:38:21.396347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.395382    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"
Oct 29 22:38:21.399935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.399911    4860 generic.go:356] "Generic (PLEG): container finished" podID="cd9fabed-9589-46a6-843a-6fd99d1278ff" containerID="8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366" exitCode=0
Oct 29 22:38:21.400076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.399979    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" event={"ID":"cd9fabed-9589-46a6-843a-6fd99d1278ff","Type":"ContainerDied","Data":"8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366"}
Oct 29 22:38:21.400076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.399982    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"
Oct 29 22:38:21.400076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.400014    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb" event={"ID":"cd9fabed-9589-46a6-843a-6fd99d1278ff","Type":"ContainerDied","Data":"9a01e72c3badbfe9fd01d93b061f27eefbbf29ebeee873569095452fea5967d3"}
Oct 29 22:38:21.411591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.411576    4860 scope.go:122] "RemoveContainer" containerID="84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164"
Oct 29 22:38:21.415694 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:21.415419    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164\": container with ID starting with 84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164 not found: ID does not exist" containerID="84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164"
Oct 29 22:38:21.415694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.415449    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164"} err="failed to get container status \"84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164\": rpc error: code = NotFound desc = could not find container \"84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164\": container with ID starting with 84e93c91f1e2c837ba99bbf290217978ac80a6da7b423c66e4fb0c2a6b74f164 not found: ID does not exist"
Oct 29 22:38:21.415694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.415467    4860 scope.go:122] "RemoveContainer" containerID="8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366"
Oct 29 22:38:21.427372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.427302    4860 scope.go:122] "RemoveContainer" containerID="8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366"
Oct 29 22:38:21.427656 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:21.427628    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366\": container with ID starting with 8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366 not found: ID does not exist" containerID="8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366"
Oct 29 22:38:21.427771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.427661    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366"} err="failed to get container status \"8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366\": rpc error: code = NotFound desc = could not find container \"8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366\": container with ID starting with 8158733e7a9b4c08aea3e12f4726118ba3e5451190aff64a8c52e89e4851d366 not found: ID does not exist"
Oct 29 22:38:21.462621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.462593    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"]
Oct 29 22:38:21.498189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.498043    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:21.501461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.501435    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-jlnzb"]
Oct 29 22:38:21.526029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.526005    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"]
Oct 29 22:38:21.531038 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:21.531017    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1131a29d-openshift-default-54ccb47c65-mx7rd"]
Oct 29 22:38:22.115088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:22.115058    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" path="/var/lib/kubelet/pods/05b8f3bc-6cc7-44e3-b406-d278376a586a/volumes"
Oct 29 22:38:22.115629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:22.115614    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9fabed-9589-46a6-843a-6fd99d1278ff" path="/var/lib/kubelet/pods/cd9fabed-9589-46a6-843a-6fd99d1278ff/volumes"
Oct 29 22:38:23.200656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:23.200617    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:23.353707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:23.353673    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:23.571408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:23.570822    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:25.034471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:25.034440    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:26.810625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:26.810542    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:28.352708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:28.352655    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:29.259471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:29.259437    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:29.720599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:29.720532    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:30.058639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:30.058510    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:30.212660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:30.212624    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:30.336909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:30.336822    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:30.469096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:30.469063    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:31.638058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:31.638026    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:32.511142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:32.511100    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:32.609592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:32.609562    4860 generic.go:356] "Generic (PLEG): container finished" podID="551bb653-c966-43cc-8f02-007606849b1f" containerID="0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813" exitCode=0
Oct 29 22:38:32.609752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:32.609639    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" event={"ID":"551bb653-c966-43cc-8f02-007606849b1f","Type":"ContainerDied","Data":"0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813"}
Oct 29 22:38:32.610096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:32.610080    4860 scope.go:122] "RemoveContainer" containerID="0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813"
Oct 29 22:38:32.611973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:32.611945    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:38:32.612066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:32.612013    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:38:32.612066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:32.612045    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:38:32.651437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:32.651412    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:32.673074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:32.673050    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"]
Oct 29 22:38:32.694002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:32.693973    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-nb6d7"]
Oct 29 22:38:32.840115 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:32.840023    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:33.622275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:33.622235    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" event={"ID":"551bb653-c966-43cc-8f02-007606849b1f","Type":"ContainerStarted","Data":"8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa"}
Oct 29 22:38:33.805313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:33.805280    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:38:33.806524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:33.806495    4860 patch_prober.go:29] interesting pod/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 29 22:38:33.806657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:33.806546    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 29 22:38:34.115462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:34.115429    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5" path="/var/lib/kubelet/pods/e7dcccb8-b2e6-4e80-bb50-c0f0cf7c65d5/volumes"
Oct 29 22:38:34.159489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:34.159459    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:34.340349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:34.334843    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:34.548344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:34.547829    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:34.814402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:34.813836    4860 patch_prober.go:29] interesting pod/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 29 22:38:34.814402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:34.813900    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 29 22:38:34.858695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:34.858667    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:38:35.805463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:35.805431    4860 patch_prober.go:29] interesting pod/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 29 22:38:35.805645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:35.805480    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 29 22:38:35.907356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:35.907206    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:36.106397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.105366    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:36.526356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.525520    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"]
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.526870    4860 cpu_manager.go:414] "containerMap: removing container" podUID="cd9fabed-9589-46a6-843a-6fd99d1278ff" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.526894    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9fabed-9589-46a6-843a-6fd99d1278ff" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.526909    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1bb026b1-e8ce-461a-bcdc-a38955de1842" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.526918    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb026b1-e8ce-461a-bcdc-a38955de1842" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.526965    4860 cpu_manager.go:414] "containerMap: removing container" podUID="14cc6760-f3f7-45bd-9c31-655c607c4505" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.526973    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cc6760-f3f7-45bd-9c31-655c607c4505" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.527007    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9bb10043-0048-4595-bc98-b3333f888d5f" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.527015    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb10043-0048-4595-bc98-b3333f888d5f" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.527029    4860 cpu_manager.go:414] "containerMap: removing container" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.527037    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.527124    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.527132    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.527366    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9fabed-9589-46a6-843a-6fd99d1278ff" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.527384    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb026b1-e8ce-461a-bcdc-a38955de1842" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.527402    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cc6760-f3f7-45bd-9c31-655c607c4505" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.527418    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb10043-0048-4595-bc98-b3333f888d5f" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.527428    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b8f3bc-6cc7-44e3-b406-d278376a586a" containerName="istio-proxy"
Oct 29 22:38:36.528901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.527446    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e41504b-834f-4cc2-89f9-ed6b7338f644" containerName="istio-proxy"
Oct 29 22:38:36.531905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.531863    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"
Oct 29 22:38:36.537738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.536778    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"]
Oct 29 22:38:36.547997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.547973    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c1750b\""
Oct 29 22:38:36.548381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.548362    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c1750b-dockercfg-vqrvl\""
Oct 29 22:38:36.549037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.549018    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"]
Oct 29 22:38:36.557251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.555897    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"
Oct 29 22:38:36.558047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.557914    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"]
Oct 29 22:38:36.565750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.565731    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"]
Oct 29 22:38:36.617525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.617485    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy" containerID="cri-o://ab83100751b821238e38a1d033dec3713018185db3798ad191d0906cad4680e1" gracePeriod=30
Oct 29 22:38:36.619224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.619205    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"]
Oct 29 22:38:36.619663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.619638    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" podUID="fb326055-bb08-42f0-8597-ddfe942b2ca5" containerName="istio-proxy" containerID="cri-o://7544e8a2c87c255d120de64015a9dff6c4ff45e8e041bfcf3c5da2dc82068a1b" gracePeriod=30
Oct 29 22:38:36.638704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.638648    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"]
Oct 29 22:38:36.641394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.639566    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np" podUID="e77e610e-005e-4e06-a301-161d60b6df15" containerName="istio-proxy" containerID="cri-o://a93a372e64e09cb38577dcbb31f4d7b22f1089a14c885bd82ca161222083d1e9" gracePeriod=30
Oct 29 22:38:36.648879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.648852    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"]
Oct 29 22:38:36.649298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.649270    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" podUID="528f6325-67f0-490d-b4a0-cf71638282b7" containerName="istio-proxy" containerID="cri-o://d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789" gracePeriod=30
Oct 29 22:38:36.706549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.698266    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54216655-1a8f-47c4-baed-e71d9b24460f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl\" (UID: \"54216655-1a8f-47c4-baed-e71d9b24460f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"
Oct 29 22:38:36.707211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.707182    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11a66f5-937a-4e05-b359-0df3d2f43492-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn\" (UID: \"b11a66f5-937a-4e05-b359-0df3d2f43492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"
Oct 29 22:38:36.707787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.707764    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pdh4\" (UniqueName: \"kubernetes.io/projected/54216655-1a8f-47c4-baed-e71d9b24460f-kube-api-access-6pdh4\") pod \"payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl\" (UID: \"54216655-1a8f-47c4-baed-e71d9b24460f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"
Oct 29 22:38:36.707995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.707979    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp5fd\" (UniqueName: \"kubernetes.io/projected/b11a66f5-937a-4e05-b359-0df3d2f43492-kube-api-access-tp5fd\") pod \"payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn\" (UID: \"b11a66f5-937a-4e05-b359-0df3d2f43492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"
Oct 29 22:38:36.708394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.702253    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:36.820904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.820804    4860 patch_prober.go:29] interesting pod/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 29 22:38:36.821198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.821154    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 29 22:38:36.831416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.831033    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp5fd\" (UniqueName: \"kubernetes.io/projected/b11a66f5-937a-4e05-b359-0df3d2f43492-kube-api-access-tp5fd\") pod \"payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn\" (UID: \"b11a66f5-937a-4e05-b359-0df3d2f43492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"
Oct 29 22:38:36.831964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.831937    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54216655-1a8f-47c4-baed-e71d9b24460f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl\" (UID: \"54216655-1a8f-47c4-baed-e71d9b24460f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"
Oct 29 22:38:36.833141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.833118    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11a66f5-937a-4e05-b359-0df3d2f43492-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn\" (UID: \"b11a66f5-937a-4e05-b359-0df3d2f43492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"
Oct 29 22:38:36.833641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.833611    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdh4\" (UniqueName: \"kubernetes.io/projected/54216655-1a8f-47c4-baed-e71d9b24460f-kube-api-access-6pdh4\") pod \"payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl\" (UID: \"54216655-1a8f-47c4-baed-e71d9b24460f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"
Oct 29 22:38:36.834225 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.833043    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54216655-1a8f-47c4-baed-e71d9b24460f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl\" (UID: \"54216655-1a8f-47c4-baed-e71d9b24460f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"
Oct 29 22:38:36.839588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.839557    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:36.840037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.840014    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11a66f5-937a-4e05-b359-0df3d2f43492-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn\" (UID: \"b11a66f5-937a-4e05-b359-0df3d2f43492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"
Oct 29 22:38:36.860128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.859539    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp5fd\" (UniqueName: \"kubernetes.io/projected/b11a66f5-937a-4e05-b359-0df3d2f43492-kube-api-access-tp5fd\") pod \"payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn\" (UID: \"b11a66f5-937a-4e05-b359-0df3d2f43492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"
Oct 29 22:38:36.860911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.860856    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdh4\" (UniqueName: \"kubernetes.io/projected/54216655-1a8f-47c4-baed-e71d9b24460f-kube-api-access-6pdh4\") pod \"payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl\" (UID: \"54216655-1a8f-47c4-baed-e71d9b24460f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"
Oct 29 22:38:36.916111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.916076    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"
Oct 29 22:38:36.944706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:36.944668    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"
Oct 29 22:38:37.300799 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:38:37.300766    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11a66f5_937a_4e05_b359_0df3d2f43492.slice/crio-126f395014df6b16525358fb81475af9a18a1949cda835354760ecca4c804d44 WatchSource:0}: Error finding container 126f395014df6b16525358fb81475af9a18a1949cda835354760ecca4c804d44: Status 404 returned error can't find the container with id 126f395014df6b16525358fb81475af9a18a1949cda835354760ecca4c804d44
Oct 29 22:38:37.301362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:37.301339    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"]
Oct 29 22:38:37.354092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:37.354069    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"]
Oct 29 22:38:37.722747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:37.722712    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl" event={"ID":"54216655-1a8f-47c4-baed-e71d9b24460f","Type":"ContainerStarted","Data":"82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888"}
Oct 29 22:38:37.722957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:37.722776    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl" event={"ID":"54216655-1a8f-47c4-baed-e71d9b24460f","Type":"ContainerStarted","Data":"a0f4f2e6f14871410bdade232354649d67027b2c0b2d76ef095483513e31f610"}
Oct 29 22:38:37.722957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:37.722808    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"
Oct 29 22:38:37.724505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:37.724476    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn" event={"ID":"b11a66f5-937a-4e05-b359-0df3d2f43492","Type":"ContainerStarted","Data":"d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86"}
Oct 29 22:38:37.724627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:37.724520    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn" event={"ID":"b11a66f5-937a-4e05-b359-0df3d2f43492","Type":"ContainerStarted","Data":"126f395014df6b16525358fb81475af9a18a1949cda835354760ecca4c804d44"}
Oct 29 22:38:37.724627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:37.724541    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"
Oct 29 22:38:37.792351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:37.791001    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl" podStartSLOduration=1.7909819869999999 podStartE2EDuration="1.790981987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:37.764310953 +0000 UTC m=+1459.789087075" watchObservedRunningTime="2026-10-29 22:38:37.790981987 +0000 UTC m=+1459.815758120"
Oct 29 22:38:37.805793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:37.805765    4860 patch_prober.go:29] interesting pod/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 29 22:38:37.805926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:37.805812    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 29 22:38:37.812941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:37.812893    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn" podStartSLOduration=1.812881612 podStartE2EDuration="1.812881612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:37.808150481 +0000 UTC m=+1459.832926612" watchObservedRunningTime="2026-10-29 22:38:37.812881612 +0000 UTC m=+1459.837657742"
Oct 29 22:38:38.396465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:38.396424    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:38.805537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:38.805500    4860 patch_prober.go:29] interesting pod/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Oct 29 22:38:38.805736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:38.805550    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Oct 29 22:38:39.811829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:39.811798    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:38:39.813238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:39.813205    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:38:40.166147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:40.166047    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:40.287915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:40.287872    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:41.144171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.144133    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:41.274874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.274842    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:41.298219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.298189    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"]
Oct 29 22:38:41.303162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.303144    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"
Oct 29 22:38:41.323253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.323224    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"]
Oct 29 22:38:41.329404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.329381    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1fa4b8b6-dockercfg-mlcnq\""
Oct 29 22:38:41.329508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.329398    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1fa4b8b6\""
Oct 29 22:38:41.337657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.337631    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/099d59a7-42df-4429-a904-61fb44a67d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf\" (UID: \"099d59a7-42df-4429-a904-61fb44a67d79\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"
Oct 29 22:38:41.337961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.337932    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfl9n\" (UniqueName: \"kubernetes.io/projected/099d59a7-42df-4429-a904-61fb44a67d79-kube-api-access-tfl9n\") pod \"payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf\" (UID: \"099d59a7-42df-4429-a904-61fb44a67d79\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"
Oct 29 22:38:41.496347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.488598    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/099d59a7-42df-4429-a904-61fb44a67d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf\" (UID: \"099d59a7-42df-4429-a904-61fb44a67d79\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"
Oct 29 22:38:41.496347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.488942    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfl9n\" (UniqueName: \"kubernetes.io/projected/099d59a7-42df-4429-a904-61fb44a67d79-kube-api-access-tfl9n\") pod \"payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf\" (UID: \"099d59a7-42df-4429-a904-61fb44a67d79\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"
Oct 29 22:38:41.496347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.490441    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/099d59a7-42df-4429-a904-61fb44a67d79-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf\" (UID: \"099d59a7-42df-4429-a904-61fb44a67d79\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"
Oct 29 22:38:41.530139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.530106    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfl9n\" (UniqueName: \"kubernetes.io/projected/099d59a7-42df-4429-a904-61fb44a67d79-kube-api-access-tfl9n\") pod \"payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf\" (UID: \"099d59a7-42df-4429-a904-61fb44a67d79\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"
Oct 29 22:38:41.533586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.533541    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"]
Oct 29 22:38:41.560503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.558278    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"]
Oct 29 22:38:41.560503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.558452    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"
Oct 29 22:38:41.603247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.592495    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk2sw\" (UniqueName: \"kubernetes.io/projected/46dfd5e5-280c-4064-95cf-1b3162c50cb2-kube-api-access-pk2sw\") pod \"payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg\" (UID: \"46dfd5e5-280c-4064-95cf-1b3162c50cb2\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"
Oct 29 22:38:41.603247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.592695    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46dfd5e5-280c-4064-95cf-1b3162c50cb2-plugins-config-volume\") pod \"payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg\" (UID: \"46dfd5e5-280c-4064-95cf-1b3162c50cb2\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"
Oct 29 22:38:41.632370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.620925    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"
Oct 29 22:38:41.703222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.703181    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk2sw\" (UniqueName: \"kubernetes.io/projected/46dfd5e5-280c-4064-95cf-1b3162c50cb2-kube-api-access-pk2sw\") pod \"payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg\" (UID: \"46dfd5e5-280c-4064-95cf-1b3162c50cb2\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"
Oct 29 22:38:41.703757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.703730    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46dfd5e5-280c-4064-95cf-1b3162c50cb2-plugins-config-volume\") pod \"payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg\" (UID: \"46dfd5e5-280c-4064-95cf-1b3162c50cb2\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"
Oct 29 22:38:41.705819 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.705768    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46dfd5e5-280c-4064-95cf-1b3162c50cb2-plugins-config-volume\") pod \"payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg\" (UID: \"46dfd5e5-280c-4064-95cf-1b3162c50cb2\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"
Oct 29 22:38:41.740286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.740211    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk2sw\" (UniqueName: \"kubernetes.io/projected/46dfd5e5-280c-4064-95cf-1b3162c50cb2-kube-api-access-pk2sw\") pod \"payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg\" (UID: \"46dfd5e5-280c-4064-95cf-1b3162c50cb2\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"
Oct 29 22:38:41.772947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.772896    4860 generic.go:356] "Generic (PLEG): container finished" podID="e77e610e-005e-4e06-a301-161d60b6df15" containerID="a93a372e64e09cb38577dcbb31f4d7b22f1089a14c885bd82ca161222083d1e9" exitCode=0
Oct 29 22:38:41.773186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.773162    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np" event={"ID":"e77e610e-005e-4e06-a301-161d60b6df15","Type":"ContainerDied","Data":"a93a372e64e09cb38577dcbb31f4d7b22f1089a14c885bd82ca161222083d1e9"}
Oct 29 22:38:41.793581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.793552    4860 generic.go:356] "Generic (PLEG): container finished" podID="55aabe95-a731-4377-966e-238d39d5ea99" containerID="ab83100751b821238e38a1d033dec3713018185db3798ad191d0906cad4680e1" exitCode=0
Oct 29 22:38:41.793712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.793664    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" event={"ID":"55aabe95-a731-4377-966e-238d39d5ea99","Type":"ContainerDied","Data":"ab83100751b821238e38a1d033dec3713018185db3798ad191d0906cad4680e1"}
Oct 29 22:38:41.796037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.796009    4860 generic.go:356] "Generic (PLEG): container finished" podID="fb326055-bb08-42f0-8597-ddfe942b2ca5" containerID="7544e8a2c87c255d120de64015a9dff6c4ff45e8e041bfcf3c5da2dc82068a1b" exitCode=0
Oct 29 22:38:41.796158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.796115    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" event={"ID":"fb326055-bb08-42f0-8597-ddfe942b2ca5","Type":"ContainerDied","Data":"7544e8a2c87c255d120de64015a9dff6c4ff45e8e041bfcf3c5da2dc82068a1b"}
Oct 29 22:38:41.911231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.911129    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"
Oct 29 22:38:41.984213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:41.983906    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:38:42.014060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.014036    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-podinfo\") pod \"fb326055-bb08-42f0-8597-ddfe942b2ca5\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") "
Oct 29 22:38:42.014170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.014130    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-kube-api-access-hlqr9\" (UniqueName: \"kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-kube-api-access-hlqr9\") pod \"fb326055-bb08-42f0-8597-ddfe942b2ca5\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") "
Oct 29 22:38:42.014616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.014252    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-socket\") pod \"fb326055-bb08-42f0-8597-ddfe942b2ca5\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") "
Oct 29 22:38:42.014616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.014406    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-data\") pod \"fb326055-bb08-42f0-8597-ddfe942b2ca5\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") "
Oct 29 22:38:42.014616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.014442    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb326055-bb08-42f0-8597-ddfe942b2ca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb326055-bb08-42f0-8597-ddfe942b2ca5-istiod-ca-cert\") pod \"fb326055-bb08-42f0-8597-ddfe942b2ca5\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") "
Oct 29 22:38:42.014616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.014507    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-token\" (UniqueName: \"kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-token\") pod \"fb326055-bb08-42f0-8597-ddfe942b2ca5\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") "
Oct 29 22:38:42.014616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.014536    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-credential-socket\") pod \"fb326055-bb08-42f0-8597-ddfe942b2ca5\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") "
Oct 29 22:38:42.014980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.014700    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-envoy\") pod \"fb326055-bb08-42f0-8597-ddfe942b2ca5\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") "
Oct 29 22:38:42.014980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.014743    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-certs\") pod \"fb326055-bb08-42f0-8597-ddfe942b2ca5\" (UID: \"fb326055-bb08-42f0-8597-ddfe942b2ca5\") "
Oct 29 22:38:42.015081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.015061    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-data" pod "fb326055-bb08-42f0-8597-ddfe942b2ca5" (UID: "fb326055-bb08-42f0-8597-ddfe942b2ca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.015272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.015248    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-socket" pod "fb326055-bb08-42f0-8597-ddfe942b2ca5" (UID: "fb326055-bb08-42f0-8597-ddfe942b2ca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.016105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.015379    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-credential-socket" pod "fb326055-bb08-42f0-8597-ddfe942b2ca5" (UID: "fb326055-bb08-42f0-8597-ddfe942b2ca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.016105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.015687    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb326055-bb08-42f0-8597-ddfe942b2ca5-istiod-ca-cert" pod "fb326055-bb08-42f0-8597-ddfe942b2ca5" (UID: "fb326055-bb08-42f0-8597-ddfe942b2ca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:42.016105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.015997    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.016105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.016018    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.016105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.016035    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb326055-bb08-42f0-8597-ddfe942b2ca5-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.016105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.016049    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.016105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.016065    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-certs" pod "fb326055-bb08-42f0-8597-ddfe942b2ca5" (UID: "fb326055-bb08-42f0-8597-ddfe942b2ca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.020260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.020023    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-envoy" pod "fb326055-bb08-42f0-8597-ddfe942b2ca5" (UID: "fb326055-bb08-42f0-8597-ddfe942b2ca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.020938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.020768    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-podinfo" pod "fb326055-bb08-42f0-8597-ddfe942b2ca5" (UID: "fb326055-bb08-42f0-8597-ddfe942b2ca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:42.020938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.020911    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-token" pod "fb326055-bb08-42f0-8597-ddfe942b2ca5" (UID: "fb326055-bb08-42f0-8597-ddfe942b2ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:42.022671 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.022636    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-kube-api-access-hlqr9" pod "fb326055-bb08-42f0-8597-ddfe942b2ca5" (UID: "fb326055-bb08-42f0-8597-ddfe942b2ca5"). InnerVolumeSpecName "kube-api-access-hlqr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:42.098914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.098880    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:38:42.115712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.115683    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:38:42.118499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.118470    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.118633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.118502    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlqr9\" (UniqueName: \"kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-kube-api-access-hlqr9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.118633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.118518    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.118633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.118532    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.118633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.118544    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb326055-bb08-42f0-8597-ddfe942b2ca5-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.219937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.219215    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-certs\") pod \"528f6325-67f0-490d-b4a0-cf71638282b7\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") "
Oct 29 22:38:42.219937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.219268    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-data\") pod \"55aabe95-a731-4377-966e-238d39d5ea99\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") "
Oct 29 22:38:42.219937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.219377    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-socket\") pod \"528f6325-67f0-490d-b4a0-cf71638282b7\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") "
Oct 29 22:38:42.219937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.219418    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55aabe95-a731-4377-966e-238d39d5ea99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aabe95-a731-4377-966e-238d39d5ea99-istio-podinfo\") pod \"55aabe95-a731-4377-966e-238d39d5ea99\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") "
Oct 29 22:38:42.219937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.219487    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55aabe95-a731-4377-966e-238d39d5ea99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aabe95-a731-4377-966e-238d39d5ea99-istiod-ca-cert\") pod \"55aabe95-a731-4377-966e-238d39d5ea99\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") "
Oct 29 22:38:42.219937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.219563    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-certs\") pod \"55aabe95-a731-4377-966e-238d39d5ea99\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") "
Oct 29 22:38:42.219937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.219604    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-data\") pod \"528f6325-67f0-490d-b4a0-cf71638282b7\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") "
Oct 29 22:38:42.219937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.219654    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-certs" pod "528f6325-67f0-490d-b4a0-cf71638282b7" (UID: "528f6325-67f0-490d-b4a0-cf71638282b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.219937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.219667    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-credential-socket\") pod \"55aabe95-a731-4377-966e-238d39d5ea99\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") "
Oct 29 22:38:42.219937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.219757    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-istio-token\" (UniqueName: \"kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-istio-token\") pod \"528f6325-67f0-490d-b4a0-cf71638282b7\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") "
Oct 29 22:38:42.219937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.219911    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-credential-socket" pod "55aabe95-a731-4377-966e-238d39d5ea99" (UID: "55aabe95-a731-4377-966e-238d39d5ea99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.220885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.220110    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-certs" pod "55aabe95-a731-4377-966e-238d39d5ea99" (UID: "55aabe95-a731-4377-966e-238d39d5ea99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.220885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.220149    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-credential-socket\") pod \"528f6325-67f0-490d-b4a0-cf71638282b7\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") "
Oct 29 22:38:42.220885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.220187    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-envoy\") pod \"55aabe95-a731-4377-966e-238d39d5ea99\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") "
Oct 29 22:38:42.220885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.220218    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-kube-api-access-wl2xd\" (UniqueName: \"kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-kube-api-access-wl2xd\") pod \"55aabe95-a731-4377-966e-238d39d5ea99\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") "
Oct 29 22:38:42.220885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.220297    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-kube-api-access-7gqh6\" (UniqueName: \"kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-kube-api-access-7gqh6\") pod \"528f6325-67f0-490d-b4a0-cf71638282b7\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") "
Oct 29 22:38:42.220885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.220367    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528f6325-67f0-490d-b4a0-cf71638282b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528f6325-67f0-490d-b4a0-cf71638282b7-istiod-ca-cert\") pod \"528f6325-67f0-490d-b4a0-cf71638282b7\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") "
Oct 29 22:38:42.220885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.220460    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-istio-token\" (UniqueName: \"kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-istio-token\") pod \"55aabe95-a731-4377-966e-238d39d5ea99\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") "
Oct 29 22:38:42.220885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.220464    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-data" pod "528f6325-67f0-490d-b4a0-cf71638282b7" (UID: "528f6325-67f0-490d-b4a0-cf71638282b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.220885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.220492    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/528f6325-67f0-490d-b4a0-cf71638282b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528f6325-67f0-490d-b4a0-cf71638282b7-istio-podinfo\") pod \"528f6325-67f0-490d-b4a0-cf71638282b7\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") "
Oct 29 22:38:42.220885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.220808    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-credential-socket" pod "528f6325-67f0-490d-b4a0-cf71638282b7" (UID: "528f6325-67f0-490d-b4a0-cf71638282b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.221467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.221233    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55aabe95-a731-4377-966e-238d39d5ea99-istiod-ca-cert" pod "55aabe95-a731-4377-966e-238d39d5ea99" (UID: "55aabe95-a731-4377-966e-238d39d5ea99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:42.221809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.221781    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528f6325-67f0-490d-b4a0-cf71638282b7-istiod-ca-cert" pod "528f6325-67f0-490d-b4a0-cf71638282b7" (UID: "528f6325-67f0-490d-b4a0-cf71638282b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:42.222535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.222509    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-data" pod "55aabe95-a731-4377-966e-238d39d5ea99" (UID: "55aabe95-a731-4377-966e-238d39d5ea99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.223027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.223001    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-socket" pod "528f6325-67f0-490d-b4a0-cf71638282b7" (UID: "528f6325-67f0-490d-b4a0-cf71638282b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.223130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.223065    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-envoy\") pod \"528f6325-67f0-490d-b4a0-cf71638282b7\" (UID: \"528f6325-67f0-490d-b4a0-cf71638282b7\") "
Oct 29 22:38:42.223193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.223177    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-socket\") pod \"55aabe95-a731-4377-966e-238d39d5ea99\" (UID: \"55aabe95-a731-4377-966e-238d39d5ea99\") "
Oct 29 22:38:42.224287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.224261    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528f6325-67f0-490d-b4a0-cf71638282b7-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.224428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.224291    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.224428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.224305    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.224428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.224365    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.224428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.224383    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55aabe95-a731-4377-966e-238d39d5ea99-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.224428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.224400    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.224428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.224413    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.224428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.224428    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.224666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.224442    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.225114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.225084    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-istio-token" pod "528f6325-67f0-490d-b4a0-cf71638282b7" (UID: "528f6325-67f0-490d-b4a0-cf71638282b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:42.225605 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.225579    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-socket" pod "55aabe95-a731-4377-966e-238d39d5ea99" (UID: "55aabe95-a731-4377-966e-238d39d5ea99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.228610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.228588    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/528f6325-67f0-490d-b4a0-cf71638282b7-istio-podinfo" pod "528f6325-67f0-490d-b4a0-cf71638282b7" (UID: "528f6325-67f0-490d-b4a0-cf71638282b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:42.228886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.228869    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-envoy" pod "55aabe95-a731-4377-966e-238d39d5ea99" (UID: "55aabe95-a731-4377-966e-238d39d5ea99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.228953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.228939    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55aabe95-a731-4377-966e-238d39d5ea99-istio-podinfo" pod "55aabe95-a731-4377-966e-238d39d5ea99" (UID: "55aabe95-a731-4377-966e-238d39d5ea99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:42.229036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.229025    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-kube-api-access-7gqh6" pod "528f6325-67f0-490d-b4a0-cf71638282b7" (UID: "528f6325-67f0-490d-b4a0-cf71638282b7"). InnerVolumeSpecName "kube-api-access-7gqh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:42.230512 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.230470    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-istio-token" pod "55aabe95-a731-4377-966e-238d39d5ea99" (UID: "55aabe95-a731-4377-966e-238d39d5ea99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:42.231699 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.231672    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-kube-api-access-wl2xd" pod "55aabe95-a731-4377-966e-238d39d5ea99" (UID: "55aabe95-a731-4377-966e-238d39d5ea99"). InnerVolumeSpecName "kube-api-access-wl2xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:42.232305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.232287    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-envoy" pod "528f6325-67f0-490d-b4a0-cf71638282b7" (UID: "528f6325-67f0-490d-b4a0-cf71638282b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.251369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.243511    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:42.265663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.265641    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"]
Oct 29 22:38:42.327398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.327296    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"]
Oct 29 22:38:42.327651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.327603    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gqh6\" (UniqueName: \"kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-kube-api-access-7gqh6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.327811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.327775    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.327972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.327925    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528f6325-67f0-490d-b4a0-cf71638282b7-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.328131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.327946    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528f6325-67f0-490d-b4a0-cf71638282b7-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.328292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.328109    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.328292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.328262    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55aabe95-a731-4377-966e-238d39d5ea99-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.328587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.328521    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528f6325-67f0-490d-b4a0-cf71638282b7-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.328587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.328542    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55aabe95-a731-4377-966e-238d39d5ea99-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.328587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.328555    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl2xd\" (UniqueName: \"kubernetes.io/projected/55aabe95-a731-4377-966e-238d39d5ea99-kube-api-access-wl2xd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.354411 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:38:42.332578    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099d59a7_42df_4429_a904_61fb44a67d79.slice/crio-21352f407d6de50eb7d14088f6efc5c1ebd156bab5696c28dead75871061a4f6 WatchSource:0}: Error finding container 21352f407d6de50eb7d14088f6efc5c1ebd156bab5696c28dead75871061a4f6: Status 404 returned error can't find the container with id 21352f407d6de50eb7d14088f6efc5c1ebd156bab5696c28dead75871061a4f6
Oct 29 22:38:42.431379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.425981    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:38:42.531587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.531554    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-envoy\") pod \"e77e610e-005e-4e06-a301-161d60b6df15\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") "
Oct 29 22:38:42.531782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.531623    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-certs\") pod \"e77e610e-005e-4e06-a301-161d60b6df15\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") "
Oct 29 22:38:42.531782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.531659    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-kube-api-access-zkprn\" (UniqueName: \"kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-kube-api-access-zkprn\") pod \"e77e610e-005e-4e06-a301-161d60b6df15\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") "
Oct 29 22:38:42.531782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.531735    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-credential-socket\") pod \"e77e610e-005e-4e06-a301-161d60b6df15\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") "
Oct 29 22:38:42.531967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.531824    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-data\") pod \"e77e610e-005e-4e06-a301-161d60b6df15\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") "
Oct 29 22:38:42.531967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.531943    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-istio-token\" (UniqueName: \"kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-istio-token\") pod \"e77e610e-005e-4e06-a301-161d60b6df15\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") "
Oct 29 22:38:42.533252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.532091    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e77e610e-005e-4e06-a301-161d60b6df15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77e610e-005e-4e06-a301-161d60b6df15-istio-podinfo\") pod \"e77e610e-005e-4e06-a301-161d60b6df15\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") "
Oct 29 22:38:42.533252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.532145    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-socket\") pod \"e77e610e-005e-4e06-a301-161d60b6df15\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") "
Oct 29 22:38:42.533252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.532278    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77e610e-005e-4e06-a301-161d60b6df15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77e610e-005e-4e06-a301-161d60b6df15-istiod-ca-cert\") pod \"e77e610e-005e-4e06-a301-161d60b6df15\" (UID: \"e77e610e-005e-4e06-a301-161d60b6df15\") "
Oct 29 22:38:42.533252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.532350    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-certs" pod "e77e610e-005e-4e06-a301-161d60b6df15" (UID: "e77e610e-005e-4e06-a301-161d60b6df15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.533252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.532580    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-socket" pod "e77e610e-005e-4e06-a301-161d60b6df15" (UID: "e77e610e-005e-4e06-a301-161d60b6df15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.533252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.532995    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77e610e-005e-4e06-a301-161d60b6df15-istiod-ca-cert" pod "e77e610e-005e-4e06-a301-161d60b6df15" (UID: "e77e610e-005e-4e06-a301-161d60b6df15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:42.533252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.533054    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-credential-socket" pod "e77e610e-005e-4e06-a301-161d60b6df15" (UID: "e77e610e-005e-4e06-a301-161d60b6df15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.533252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.533091    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.533252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.533109    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-data" pod "e77e610e-005e-4e06-a301-161d60b6df15" (UID: "e77e610e-005e-4e06-a301-161d60b6df15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.533252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.533113    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.536757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.536697    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-istio-token" pod "e77e610e-005e-4e06-a301-161d60b6df15" (UID: "e77e610e-005e-4e06-a301-161d60b6df15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:42.536857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.536819    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-envoy" pod "e77e610e-005e-4e06-a301-161d60b6df15" (UID: "e77e610e-005e-4e06-a301-161d60b6df15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:38:42.537593 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.537558    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-kube-api-access-zkprn" pod "e77e610e-005e-4e06-a301-161d60b6df15" (UID: "e77e610e-005e-4e06-a301-161d60b6df15"). InnerVolumeSpecName "kube-api-access-zkprn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:42.539200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.539175    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e77e610e-005e-4e06-a301-161d60b6df15-istio-podinfo" pod "e77e610e-005e-4e06-a301-161d60b6df15" (UID: "e77e610e-005e-4e06-a301-161d60b6df15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:38:42.634451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.634415    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.634451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.634445    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77e610e-005e-4e06-a301-161d60b6df15-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.634643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.634459    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77e610e-005e-4e06-a301-161d60b6df15-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.634643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.634470    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.634643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.634483    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkprn\" (UniqueName: \"kubernetes.io/projected/e77e610e-005e-4e06-a301-161d60b6df15-kube-api-access-zkprn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.634643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.634494    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.634643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.634506    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77e610e-005e-4e06-a301-161d60b6df15-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:42.807252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.807167    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"
Oct 29 22:38:42.807252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.807195    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np" event={"ID":"e77e610e-005e-4e06-a301-161d60b6df15","Type":"ContainerDied","Data":"63115ed852f7a05dd3fc3de1cf7fbf3be77696a99c19a0162db3d5188bb2f2fe"}
Oct 29 22:38:42.807485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.807273    4860 scope.go:122] "RemoveContainer" containerID="a93a372e64e09cb38577dcbb31f4d7b22f1089a14c885bd82ca161222083d1e9"
Oct 29 22:38:42.809031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.809004    4860 generic.go:356] "Generic (PLEG): container finished" podID="528f6325-67f0-490d-b4a0-cf71638282b7" containerID="d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789" exitCode=0
Oct 29 22:38:42.809141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.809041    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"
Oct 29 22:38:42.809141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.809084    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" event={"ID":"528f6325-67f0-490d-b4a0-cf71638282b7","Type":"ContainerDied","Data":"d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789"}
Oct 29 22:38:42.809141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.809135    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4" event={"ID":"528f6325-67f0-490d-b4a0-cf71638282b7","Type":"ContainerDied","Data":"4ff997e4af414096320c07e56686a6cfca3f4ed3214d3988ae923dea5580b7f8"}
Oct 29 22:38:42.810863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.810844    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"
Oct 29 22:38:42.811017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.810888    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w" event={"ID":"55aabe95-a731-4377-966e-238d39d5ea99","Type":"ContainerDied","Data":"04f6092c3ba7a161c9d62ac5ec73749e2c182eea5b0490a7c381fce71f707aa1"}
Oct 29 22:38:42.814208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.814180    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf" event={"ID":"099d59a7-42df-4429-a904-61fb44a67d79","Type":"ContainerStarted","Data":"80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6"}
Oct 29 22:38:42.814366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.814228    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf" event={"ID":"099d59a7-42df-4429-a904-61fb44a67d79","Type":"ContainerStarted","Data":"21352f407d6de50eb7d14088f6efc5c1ebd156bab5696c28dead75871061a4f6"}
Oct 29 22:38:42.814366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.814258    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"
Oct 29 22:38:42.816483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.816053    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"
Oct 29 22:38:42.816483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.816059    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb" event={"ID":"fb326055-bb08-42f0-8597-ddfe942b2ca5","Type":"ContainerDied","Data":"081aa9aaf4b49e9cfc942d5b182331e877f888c78dc4a697cc6092cb034df64d"}
Oct 29 22:38:42.825139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.825119    4860 scope.go:122] "RemoveContainer" containerID="d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789"
Oct 29 22:38:42.827390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.826778    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg" event={"ID":"46dfd5e5-280c-4064-95cf-1b3162c50cb2","Type":"ContainerStarted","Data":"c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe"}
Oct 29 22:38:42.827390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.826825    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg" event={"ID":"46dfd5e5-280c-4064-95cf-1b3162c50cb2","Type":"ContainerStarted","Data":"5465394a62d59b6d5c0d3fc3c063338e6d11a84596c9e9dd5736ed9619c32f6a"}
Oct 29 22:38:42.827390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.826852    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"
Oct 29 22:38:42.838493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.838474    4860 scope.go:122] "RemoveContainer" containerID="d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789"
Oct 29 22:38:42.838751 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:42.838729    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789\": container with ID starting with d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789 not found: ID does not exist" containerID="d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789"
Oct 29 22:38:42.838800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.838759    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789"} err="failed to get container status \"d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789\": rpc error: code = NotFound desc = could not find container \"d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789\": container with ID starting with d45b4defa50862b93f76d3e400359644d22bc1428255ec66278d1ee455e60789 not found: ID does not exist"
Oct 29 22:38:42.838800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.838776    4860 scope.go:122] "RemoveContainer" containerID="ab83100751b821238e38a1d033dec3713018185db3798ad191d0906cad4680e1"
Oct 29 22:38:42.848968 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.848940    4860 scope.go:122] "RemoveContainer" containerID="7544e8a2c87c255d120de64015a9dff6c4ff45e8e041bfcf3c5da2dc82068a1b"
Oct 29 22:38:42.968990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.968932    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf" podStartSLOduration=1.96891545 podStartE2EDuration="1.96891545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:42.878264495 +0000 UTC m=+1464.903040626" watchObservedRunningTime="2026-10-29 22:38:42.96891545 +0000 UTC m=+1464.993691584"
Oct 29 22:38:42.973081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.973057    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"]
Oct 29 22:38:42.979452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:42.979429    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-h947w"]
Oct 29 22:38:43.037111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:43.037063    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"]
Oct 29 22:38:43.049160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:43.049128    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-jp7np"]
Oct 29 22:38:43.067743 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:43.067664    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"]
Oct 29 22:38:43.073903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:43.073879    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-95zbb"]
Oct 29 22:38:43.116721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:43.116661    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg" podStartSLOduration=2.116642853 podStartE2EDuration="2.116642853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:43.116068746 +0000 UTC m=+1465.140844874" watchObservedRunningTime="2026-10-29 22:38:43.116642853 +0000 UTC m=+1465.141418986"
Oct 29 22:38:43.162208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:43.162178    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"]
Oct 29 22:38:43.167076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:43.167050    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-43b37d-openshift-default-5dcf864fcc-s5cq4"]
Oct 29 22:38:44.113658 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:44.113623    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528f6325-67f0-490d-b4a0-cf71638282b7" path="/var/lib/kubelet/pods/528f6325-67f0-490d-b4a0-cf71638282b7/volumes"
Oct 29 22:38:44.114204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:44.114188    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55aabe95-a731-4377-966e-238d39d5ea99" path="/var/lib/kubelet/pods/55aabe95-a731-4377-966e-238d39d5ea99/volumes"
Oct 29 22:38:44.114793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:44.114778    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77e610e-005e-4e06-a301-161d60b6df15" path="/var/lib/kubelet/pods/e77e610e-005e-4e06-a301-161d60b6df15/volumes"
Oct 29 22:38:44.115634 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:44.115618    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb326055-bb08-42f0-8597-ddfe942b2ca5" path="/var/lib/kubelet/pods/fb326055-bb08-42f0-8597-ddfe942b2ca5/volumes"
Oct 29 22:38:44.224778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:44.224743    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:46.768335 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:46.768301    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:46.886756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:46.886708    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:47.855781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.855749    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:47.918189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.918155    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"]
Oct 29 22:38:47.919394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.919365    4860 cpu_manager.go:414] "containerMap: removing container" podUID="fb326055-bb08-42f0-8597-ddfe942b2ca5" containerName="istio-proxy"
Oct 29 22:38:47.919394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.919390    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb326055-bb08-42f0-8597-ddfe942b2ca5" containerName="istio-proxy"
Oct 29 22:38:47.919394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.919400    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e77e610e-005e-4e06-a301-161d60b6df15" containerName="istio-proxy"
Oct 29 22:38:47.919674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.919405    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77e610e-005e-4e06-a301-161d60b6df15" containerName="istio-proxy"
Oct 29 22:38:47.919674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.919448    4860 cpu_manager.go:414] "containerMap: removing container" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy"
Oct 29 22:38:47.919674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.919455    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy"
Oct 29 22:38:47.919674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.919466    4860 cpu_manager.go:414] "containerMap: removing container" podUID="528f6325-67f0-490d-b4a0-cf71638282b7" containerName="istio-proxy"
Oct 29 22:38:47.919674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.919472    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528f6325-67f0-490d-b4a0-cf71638282b7" containerName="istio-proxy"
Oct 29 22:38:47.919674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.919617    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="55aabe95-a731-4377-966e-238d39d5ea99" containerName="istio-proxy"
Oct 29 22:38:47.919674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.919631    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="528f6325-67f0-490d-b4a0-cf71638282b7" containerName="istio-proxy"
Oct 29 22:38:47.919674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.919649    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb326055-bb08-42f0-8597-ddfe942b2ca5" containerName="istio-proxy"
Oct 29 22:38:47.919674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.919661    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77e610e-005e-4e06-a301-161d60b6df15" containerName="istio-proxy"
Oct 29 22:38:47.925974 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.925952    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:47.948578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.948549    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"]
Oct 29 22:38:47.951193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.951168    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:38:47.951545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:47.951526    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:38:48.057911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.057876    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw\" (UID: \"a0ffc8b5-da6b-4ced-860e-e87812cda589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:48.058077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.057918    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvfn\" (UniqueName: \"kubernetes.io/projected/a0ffc8b5-da6b-4ced-860e-e87812cda589-kube-api-access-kpvfn\") pod \"maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw\" (UID: \"a0ffc8b5-da6b-4ced-860e-e87812cda589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:48.058077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.058055    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw\" (UID: \"a0ffc8b5-da6b-4ced-860e-e87812cda589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:48.059755 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:48.059721    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:38:48.159286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.159204    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw\" (UID: \"a0ffc8b5-da6b-4ced-860e-e87812cda589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:48.159286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.159249    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvfn\" (UniqueName: \"kubernetes.io/projected/a0ffc8b5-da6b-4ced-860e-e87812cda589-kube-api-access-kpvfn\") pod \"maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw\" (UID: \"a0ffc8b5-da6b-4ced-860e-e87812cda589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:48.159545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.159315    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw\" (UID: \"a0ffc8b5-da6b-4ced-860e-e87812cda589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:48.160252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.160229    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw\" (UID: \"a0ffc8b5-da6b-4ced-860e-e87812cda589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:48.161637 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.161605    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw\" (UID: \"a0ffc8b5-da6b-4ced-860e-e87812cda589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:48.179872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.179846    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvfn\" (UniqueName: \"kubernetes.io/projected/a0ffc8b5-da6b-4ced-860e-e87812cda589-kube-api-access-kpvfn\") pod \"maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw\" (UID: \"a0ffc8b5-da6b-4ced-860e-e87812cda589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:48.290067 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.290043    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:48.594211 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:38:48.594178    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ffc8b5_da6b_4ced_860e_e87812cda589.slice/crio-8a28b71818195fab50bdc4e089de8006f2234432a8385e388b35f36897830ea9 WatchSource:0}: Error finding container 8a28b71818195fab50bdc4e089de8006f2234432a8385e388b35f36897830ea9: Status 404 returned error can't find the container with id 8a28b71818195fab50bdc4e089de8006f2234432a8385e388b35f36897830ea9
Oct 29 22:38:48.596818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.596790    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"]
Oct 29 22:38:48.738149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.738123    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"
Oct 29 22:38:48.738284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.738189    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"
Oct 29 22:38:48.973339 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:48.973294    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw" event={"ID":"a0ffc8b5-da6b-4ced-860e-e87812cda589","Type":"ContainerStarted","Data":"8a28b71818195fab50bdc4e089de8006f2234432a8385e388b35f36897830ea9"}
Oct 29 22:38:49.494700 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:49.494673    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:49.984392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:49.984361    4860 generic.go:356] "Generic (PLEG): container finished" podID="a0ffc8b5-da6b-4ced-860e-e87812cda589" containerID="820328f8dfb8251397fc38631bf22a7e7e21d21019db7443d19d5963876a037c" exitCode=0
Oct 29 22:38:49.984782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:49.984429    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw" event={"ID":"a0ffc8b5-da6b-4ced-860e-e87812cda589","Type":"ContainerDied","Data":"820328f8dfb8251397fc38631bf22a7e7e21d21019db7443d19d5963876a037c"}
Oct 29 22:38:51.174663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:51.174634    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:51.209786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:51.209758    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ffc8b5-da6b-4ced-860e-e87812cda589-kube-api-access-kpvfn\" (UniqueName: \"kubernetes.io/projected/a0ffc8b5-da6b-4ced-860e-e87812cda589-kube-api-access-kpvfn\") pod \"a0ffc8b5-da6b-4ced-860e-e87812cda589\" (UID: \"a0ffc8b5-da6b-4ced-860e-e87812cda589\") "
Oct 29 22:38:51.209934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:51.209904    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-openshift-ca\") pod \"a0ffc8b5-da6b-4ced-860e-e87812cda589\" (UID: \"a0ffc8b5-da6b-4ced-860e-e87812cda589\") "
Oct 29 22:38:51.210019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:51.209971    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-xks-ca\") pod \"a0ffc8b5-da6b-4ced-860e-e87812cda589\" (UID: \"a0ffc8b5-da6b-4ced-860e-e87812cda589\") "
Oct 29 22:38:51.210251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:51.210228    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-openshift-ca" pod "a0ffc8b5-da6b-4ced-860e-e87812cda589" (UID: "a0ffc8b5-da6b-4ced-860e-e87812cda589"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:38:51.210691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:51.210668    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:51.212031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:51.212009    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ffc8b5-da6b-4ced-860e-e87812cda589-kube-api-access-kpvfn" pod "a0ffc8b5-da6b-4ced-860e-e87812cda589" (UID: "a0ffc8b5-da6b-4ced-860e-e87812cda589"). InnerVolumeSpecName "kube-api-access-kpvfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:38:51.212139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:51.212124    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-xks-ca" pod "a0ffc8b5-da6b-4ced-860e-e87812cda589" (UID: "a0ffc8b5-da6b-4ced-860e-e87812cda589"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:38:51.308067 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:51.307979    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:51.336137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:51.324607    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpvfn\" (UniqueName: \"kubernetes.io/projected/a0ffc8b5-da6b-4ced-860e-e87812cda589-kube-api-access-kpvfn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:51.336137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:51.324657    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ffc8b5-da6b-4ced-860e-e87812cda589-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:38:52.005433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:52.005401    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"
Oct 29 22:38:52.005433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:52.005427    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw" event={"ID":"a0ffc8b5-da6b-4ced-860e-e87812cda589","Type":"ContainerDied","Data":"8a28b71818195fab50bdc4e089de8006f2234432a8385e388b35f36897830ea9"}
Oct 29 22:38:52.005631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:52.005476    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a28b71818195fab50bdc4e089de8006f2234432a8385e388b35f36897830ea9"
Oct 29 22:38:53.033399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:53.033369    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:53.912092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:53.912058    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"
Oct 29 22:38:53.912267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:53.912147    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"
Oct 29 22:38:54.528164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.528127    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"]
Oct 29 22:38:54.528941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.528924    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a0ffc8b5-da6b-4ced-860e-e87812cda589" containerName="revoke-keys"
Oct 29 22:38:54.528983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.528942    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ffc8b5-da6b-4ced-860e-e87812cda589" containerName="revoke-keys"
Oct 29 22:38:54.529102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.529093    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ffc8b5-da6b-4ced-860e-e87812cda589" containerName="revoke-keys"
Oct 29 22:38:54.533235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.533213    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:54.560670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.560640    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4d3168-serving-cert\""
Oct 29 22:38:54.566400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.564484    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"]
Oct 29 22:38:54.625717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.625683    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-metrics-certs\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:54.625922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.625878    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:54.626000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.625973    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2jrm\" (UniqueName: \"kubernetes.io/projected/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-kube-api-access-c2jrm\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:54.626055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.626009    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:54.634612 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.634578    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:54.763350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.762845    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:54.801407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.780443    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-metrics-certs\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:54.801407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.794492    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:54.801407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.795569    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:54.801407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.795704    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2jrm\" (UniqueName: \"kubernetes.io/projected/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-kube-api-access-c2jrm\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:54.801407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.795771    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:54.801407 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:54.796608    4860 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-serving-cert: secret "maas-api-e2e-inf-a-4d3168-serving-cert" not found
Oct 29 22:38:54.801407 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:38:54.796724    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-maas-api-tls podName:c5f2142b-4344-47d4-bdf0-40a45e61bb3e nodeName:}" failed. No retries permitted until 2026-10-29 22:38:55.296705012 +0000 UTC m=+1477.321481128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-maas-api-tls") pod "maas-api-e2e-inf-a-4d3168-5785964b99-wntgm" (UID: "c5f2142b-4344-47d4-bdf0-40a45e61bb3e") : secret "maas-api-e2e-inf-a-4d3168-serving-cert" not found
Oct 29 22:38:54.831348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.818220    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-metrics-certs\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:54.854347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:54.834271    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2jrm\" (UniqueName: \"kubernetes.io/projected/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-kube-api-access-c2jrm\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:55.312715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:55.312679    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:55.315439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:55.315406    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d3168-5785964b99-wntgm\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:55.322010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:55.321971    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:55.465381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:55.465314    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:56.442709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:56.442671    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"]
Oct 29 22:38:56.662576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:56.662518    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:57.329603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:57.329512    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm" event={"ID":"c5f2142b-4344-47d4-bdf0-40a45e61bb3e","Type":"ContainerStarted","Data":"dd780bab27632d5ed82cf66fc9a35f0820c84588dfdfdea198fdb083d96494b4"}
Oct 29 22:38:58.018877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.009441    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:38:58.018877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.009998    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:38:58.018877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.016388    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:38:58.018877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.016685    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:38:58.018877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.016986    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:58.020004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.019722    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:58.043379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.040730    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:58.060398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.054751    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:38:58.084557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.084191    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:58.248390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.248347    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:58.383518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.383417    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:58.425878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.425829    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:58.426799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.426767    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm" event={"ID":"c5f2142b-4344-47d4-bdf0-40a45e61bb3e","Type":"ContainerStarted","Data":"7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395"}
Oct 29 22:38:58.428177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.428156    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:38:58.527081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.527017    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm" podStartSLOduration=4.105020566 podStartE2EDuration="4.526993866s" totalImagesPullingTime="421.9733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:58.491234619 +0000 UTC m=+1480.516010787" watchObservedRunningTime="2026-10-29 22:38:58.526993866 +0000 UTC m=+1480.551769998"
Oct 29 22:38:58.586001 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:58.585963    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:59.028350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.012393    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:38:59.028350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.012745    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:38:59.028350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.013384    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:38:59.028350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.013442    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:38:59.028350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.026942    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:59.028350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.027207    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:38:59.028350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.027257    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:38:59.028350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.027390    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": context deadline exceeded"
Oct 29 22:38:59.029308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.028782    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:38:59.029308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.028856    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:38:59.029308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.028910    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:38:59.029308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.028959    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:38:59.044557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.044518    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:59.047039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.046345    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:38:59.060936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.060908    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:59.063305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.063282    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:38:59.143348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.117975    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:59.231350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.194312    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:59.306445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.283699    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:38:59.306445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.269659    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:38:59.338380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.308479    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:38:59.410345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.410020    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Oct 29 22:38:59.423349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.411233    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:38:59.574413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.550586    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:38:59.800370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:38:59.632456    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main" probeResult="failure" output="Get \"https://10.132.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.046157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.046114    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": context deadline exceeded"
Oct 29 22:39:00.046785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.046758    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:39:00.047964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.047936    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:39:00.100552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.100504    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.137536 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.137487    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.240359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.200142    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:39:00.300405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.297725    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.450001 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.449945    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:39:00.580435 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.568449    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.622795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.622748    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.623128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.623089    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.691624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.691571    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.818932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.818884    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Oct 29 22:39:00.946406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.809440    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main" probeResult="failure" output="Get \"https://10.132.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:00.947149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:00.839990    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:01.116350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.095259    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:01.159735 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:39:01.159702    4860 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.056s"
Oct 29 22:39:01.160289 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.160273    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:39:01.160715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.160692    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 22:39:01.160828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.160745    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:39:01.160828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.160785    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:39:01.160947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.160848    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:39:01.160947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.160889    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:39:01.160947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.160921    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:39:01.161102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.160954    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 22:39:01.161102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.160986    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:39:01.161102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.161031    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m"
Oct 29 22:39:01.161102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.161063    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:39:01.161102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.161093    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:39:01.163376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:01.163355    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:39:02.039705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:02.039673    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"]
Oct 29 22:39:02.048992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:02.048959    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc893314-qkwqs"]
Oct 29 22:39:02.147757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:02.147725    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc" path="/var/lib/kubelet/pods/fdcb5a35-7e4a-43f7-98b5-05bf3b35acdc/volumes"
Oct 29 22:39:02.236981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:02.236954    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:39:02.658045 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:02.658021    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:02.790887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:02.790807    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:04.130080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.130007    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"]
Oct 29 22:39:04.136265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.136241    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:04.156049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.153417    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"]
Oct 29 22:39:04.157603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.157581    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:39:04.158162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.158138    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:39:04.168981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.168951    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-616da7-nx5ns\" (UID: \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:04.169370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.169125    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-616da7-nx5ns\" (UID: \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:04.169370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.169233    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pbvv\" (UniqueName: \"kubernetes.io/projected/2a600cd8-e011-46ab-9b3b-3f392d2afa19-kube-api-access-4pbvv\") pod \"maas-api-revoke-keys-e2e-api-a-616da7-nx5ns\" (UID: \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:04.270913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.270871    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-616da7-nx5ns\" (UID: \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:04.271119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.270956    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbvv\" (UniqueName: \"kubernetes.io/projected/2a600cd8-e011-46ab-9b3b-3f392d2afa19-kube-api-access-4pbvv\") pod \"maas-api-revoke-keys-e2e-api-a-616da7-nx5ns\" (UID: \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:04.271207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.271185    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-616da7-nx5ns\" (UID: \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:04.272104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.272079    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-616da7-nx5ns\" (UID: \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:04.274223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.274194    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-616da7-nx5ns\" (UID: \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:04.291384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.291040    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:04.293536 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.293514    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pbvv\" (UniqueName: \"kubernetes.io/projected/2a600cd8-e011-46ab-9b3b-3f392d2afa19-kube-api-access-4pbvv\") pod \"maas-api-revoke-keys-e2e-api-a-616da7-nx5ns\" (UID: \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:04.480407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.480343    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:04.812164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:04.812137    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"]
Oct 29 22:39:05.275757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:05.275694    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns" event={"ID":"2a600cd8-e011-46ab-9b3b-3f392d2afa19","Type":"ContainerStarted","Data":"25a30dd7f9d28af72071c8056f997d4019b4bfdf80909231aff6da05533b7563"}
Oct 29 22:39:05.276242 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:05.275778    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns" event={"ID":"2a600cd8-e011-46ab-9b3b-3f392d2afa19","Type":"ContainerStarted","Data":"b4a5328339f32b02bd9f6284fe00dd6612d5c9f2d570cbc01865683a412f990d"}
Oct 29 22:39:05.319558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:05.319505    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns" podStartSLOduration=1.319489937 podStartE2EDuration="1.319489937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:05.318529107 +0000 UTC m=+1487.343305239" watchObservedRunningTime="2026-10-29 22:39:05.319489937 +0000 UTC m=+1487.344266067"
Oct 29 22:39:05.949531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:05.949477    4860 patch_prober.go:29] interesting pod/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:39:05.949763 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:05.949559    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:39:06.038720 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:06.038615    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:06.289932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:06.289840    4860 generic.go:356] "Generic (PLEG): container finished" podID="2a600cd8-e011-46ab-9b3b-3f392d2afa19" containerID="25a30dd7f9d28af72071c8056f997d4019b4bfdf80909231aff6da05533b7563" exitCode=0
Oct 29 22:39:06.290317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:06.289921    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns" event={"ID":"2a600cd8-e011-46ab-9b3b-3f392d2afa19","Type":"ContainerDied","Data":"25a30dd7f9d28af72071c8056f997d4019b4bfdf80909231aff6da05533b7563"}
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.538164    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.635720    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-openshift-ca\") pod \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\" (UID: \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\") "
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.635924    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a600cd8-e011-46ab-9b3b-3f392d2afa19-kube-api-access-4pbvv\" (UniqueName: \"kubernetes.io/projected/2a600cd8-e011-46ab-9b3b-3f392d2afa19-kube-api-access-4pbvv\") pod \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\" (UID: \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\") "
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.635964    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-xks-ca\") pod \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\" (UID: \"2a600cd8-e011-46ab-9b3b-3f392d2afa19\") "
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.636112    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-openshift-ca" pod "2a600cd8-e011-46ab-9b3b-3f392d2afa19" (UID: "2a600cd8-e011-46ab-9b3b-3f392d2afa19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.636969    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.640407    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a600cd8-e011-46ab-9b3b-3f392d2afa19-kube-api-access-4pbvv" pod "2a600cd8-e011-46ab-9b3b-3f392d2afa19" (UID: "2a600cd8-e011-46ab-9b3b-3f392d2afa19"). InnerVolumeSpecName "kube-api-access-4pbvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.641037    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-xks-ca" pod "2a600cd8-e011-46ab-9b3b-3f392d2afa19" (UID: "2a600cd8-e011-46ab-9b3b-3f392d2afa19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.719697    4860 patch_prober.go:29] interesting pod/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.719765    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.759802    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pbvv\" (UniqueName: \"kubernetes.io/projected/2a600cd8-e011-46ab-9b3b-3f392d2afa19-kube-api-access-4pbvv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.759832    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a600cd8-e011-46ab-9b3b-3f392d2afa19-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.866898    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"]
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.867981    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" podUID="60189037-441a-4cb7-866c-6287bda16518" containerName="istio-proxy" containerID="cri-o://25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e" gracePeriod=30
Oct 29 22:39:08.068078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:07.994175    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:08.118367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.118180    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:08.315081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.315053    4860 generic.go:356] "Generic (PLEG): container finished" podID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerID="86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493" exitCode=0
Oct 29 22:39:08.315206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.315129    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" event={"ID":"7a537d76-6b6c-4ae9-a18d-1556b9d26625","Type":"ContainerDied","Data":"86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493"}
Oct 29 22:39:08.315677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.315655    4860 scope.go:122] "RemoveContainer" containerID="86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493"
Oct 29 22:39:08.317122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.317099    4860 generic.go:356] "Generic (PLEG): container finished" podID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerID="8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a" exitCode=0
Oct 29 22:39:08.317269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.317163    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" event={"ID":"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7","Type":"ContainerDied","Data":"8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a"}
Oct 29 22:39:08.317649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.317624    4860 scope.go:122] "RemoveContainer" containerID="8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a"
Oct 29 22:39:08.317792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.317758    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:39:08.317868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.317850    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:39:08.317931 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.317903    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:39:08.319301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.319280    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns" event={"ID":"2a600cd8-e011-46ab-9b3b-3f392d2afa19","Type":"ContainerDied","Data":"b4a5328339f32b02bd9f6284fe00dd6612d5c9f2d570cbc01865683a412f990d"}
Oct 29 22:39:08.319433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.319366    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"
Oct 29 22:39:08.319524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.319461    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:39:08.319524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.319366    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4a5328339f32b02bd9f6284fe00dd6612d5c9f2d570cbc01865683a412f990d"
Oct 29 22:39:08.319638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.319532    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:39:08.319638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.319583    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:39:08.320702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:08.320582    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:09.440347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:09.435710    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" event={"ID":"7a537d76-6b6c-4ae9-a18d-1556b9d26625","Type":"ContainerStarted","Data":"ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d"}
Oct 29 22:39:09.443218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:09.441137    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" event={"ID":"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7","Type":"ContainerStarted","Data":"62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d"}
Oct 29 22:39:09.840548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:09.837837    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:09.898670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:09.897854    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:39:09.898670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:09.898030    4860 patch_prober.go:29] interesting pod/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 29 22:39:09.898670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:09.898072    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 29 22:39:09.960654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:09.960611    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:10.334899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:10.334865    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:39:10.335122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:10.335099    4860 patch_prober.go:29] interesting pod/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:39:10.335185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:10.335149    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 29 22:39:10.911500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:10.902643    4860 patch_prober.go:29] interesting pod/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 29 22:39:10.911500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:10.902704    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 29 22:39:11.035496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.035466    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"]
Oct 29 22:39:11.036258 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.036242    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2a600cd8-e011-46ab-9b3b-3f392d2afa19" containerName="revoke-keys"
Oct 29 22:39:11.036258 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.036258    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a600cd8-e011-46ab-9b3b-3f392d2afa19" containerName="revoke-keys"
Oct 29 22:39:11.036443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.036432    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a600cd8-e011-46ab-9b3b-3f392d2afa19" containerName="revoke-keys"
Oct 29 22:39:11.040419 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.040398    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:11.060262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.060239    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"]
Oct 29 22:39:11.064726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.064170    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:39:11.064726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.064294    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:39:11.132516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.132482    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94sjb\" (UniqueName: \"kubernetes.io/projected/18e7ee26-39f3-4475-86ed-04a65e3e4f33-kube-api-access-94sjb\") pod \"maas-api-revoke-keys-e2e-api-a-ee886f-49btn\" (UID: \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:11.133013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.132989    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee886f-49btn\" (UID: \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:11.133167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.133149    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee886f-49btn\" (UID: \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:11.234586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.234554    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94sjb\" (UniqueName: \"kubernetes.io/projected/18e7ee26-39f3-4475-86ed-04a65e3e4f33-kube-api-access-94sjb\") pod \"maas-api-revoke-keys-e2e-api-a-ee886f-49btn\" (UID: \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:11.234905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.234889    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee886f-49btn\" (UID: \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:11.235055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.235037    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee886f-49btn\" (UID: \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:11.235700 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.235671    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee886f-49btn\" (UID: \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:11.237840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.237815    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee886f-49btn\" (UID: \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:11.256155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.256136    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94sjb\" (UniqueName: \"kubernetes.io/projected/18e7ee26-39f3-4475-86ed-04a65e3e4f33-kube-api-access-94sjb\") pod \"maas-api-revoke-keys-e2e-api-a-ee886f-49btn\" (UID: \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:11.334506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.334474    4860 patch_prober.go:29] interesting pod/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:39:11.334664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.334524    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 29 22:39:11.359824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.359794    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:11.712824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.712730    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:11.825490 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:39:11.824223    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e7ee26_39f3_4475_86ed_04a65e3e4f33.slice/crio-82365b1767ba1712f54d8621399eaadf6db7f6ce57690d733e4b22de21891cdc WatchSource:0}: Error finding container 82365b1767ba1712f54d8621399eaadf6db7f6ce57690d733e4b22de21891cdc: Status 404 returned error can't find the container with id 82365b1767ba1712f54d8621399eaadf6db7f6ce57690d733e4b22de21891cdc
Oct 29 22:39:11.849587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.831891    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"]
Oct 29 22:39:11.908346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.891452    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:11.928690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.922710    4860 patch_prober.go:29] interesting pod/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 29 22:39:11.928690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:11.922764    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 29 22:39:12.024834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:12.024787    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:12.045973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:12.045946    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"]
Oct 29 22:39:12.051625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:12.051600    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418d5d3e-2clvz"]
Oct 29 22:39:12.136111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:12.136037    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7454f3-4494-41b6-a10d-1d85ca7646f9" path="/var/lib/kubelet/pods/bc7454f3-4494-41b6-a10d-1d85ca7646f9/volumes"
Oct 29 22:39:12.164628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:12.164594    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:12.335665 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:12.335633    4860 patch_prober.go:29] interesting pod/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:39:12.335829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:12.335687    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 29 22:39:12.523184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:12.523145    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn" event={"ID":"18e7ee26-39f3-4475-86ed-04a65e3e4f33","Type":"ContainerStarted","Data":"82365b1767ba1712f54d8621399eaadf6db7f6ce57690d733e4b22de21891cdc"}
Oct 29 22:39:12.899431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:12.895743    4860 patch_prober.go:29] interesting pod/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 29 22:39:12.899431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:12.895822    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 29 22:39:13.334701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.334012    4860 patch_prober.go:29] interesting pod/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:39:13.334701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.334061    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 29 22:39:13.535945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.535912    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:39:13.537293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.537269    4860 generic.go:356] "Generic (PLEG): container finished" podID="60189037-441a-4cb7-866c-6287bda16518" containerID="25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e" exitCode=0
Oct 29 22:39:13.537417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.537370    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" event={"ID":"60189037-441a-4cb7-866c-6287bda16518","Type":"ContainerDied","Data":"25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e"}
Oct 29 22:39:13.537468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.537428    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4" event={"ID":"60189037-441a-4cb7-866c-6287bda16518","Type":"ContainerDied","Data":"3d9eda58f0fa9600175f872695043e1a0fad035993f5e18a673d1f76b54063ac"}
Oct 29 22:39:13.537468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.537455    4860 scope.go:122] "RemoveContainer" containerID="25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e"
Oct 29 22:39:13.539653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.539556    4860 generic.go:356] "Generic (PLEG): container finished" podID="18e7ee26-39f3-4475-86ed-04a65e3e4f33" containerID="fa68e5a889fcbe21588b82ecffe14f763fab9ac56d69e0b9bcf1e72d7471fe37" exitCode=0
Oct 29 22:39:13.539653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.539596    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn" event={"ID":"18e7ee26-39f3-4475-86ed-04a65e3e4f33","Type":"ContainerDied","Data":"fa68e5a889fcbe21588b82ecffe14f763fab9ac56d69e0b9bcf1e72d7471fe37"}
Oct 29 22:39:13.552887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.552376    4860 scope.go:122] "RemoveContainer" containerID="25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e"
Oct 29 22:39:13.553356 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:39:13.553164    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e\": container with ID starting with 25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e not found: ID does not exist" containerID="25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e"
Oct 29 22:39:13.553356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.553205    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e"} err="failed to get container status \"25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e\": rpc error: code = NotFound desc = could not find container \"25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e\": container with ID starting with 25fc748a9d8574f8e89dd8c89474f79ebc57842f0c62ec4334451049c470833e not found: ID does not exist"
Oct 29 22:39:13.645238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.645170    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:13.658417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.658388    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-socket\") pod \"60189037-441a-4cb7-866c-6287bda16518\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") "
Oct 29 22:39:13.658534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.658507    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60189037-441a-4cb7-866c-6287bda16518-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60189037-441a-4cb7-866c-6287bda16518-istio-podinfo\") pod \"60189037-441a-4cb7-866c-6287bda16518\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") "
Oct 29 22:39:13.658636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.658619    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-credential-socket\") pod \"60189037-441a-4cb7-866c-6287bda16518\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") "
Oct 29 22:39:13.658683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.658671    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-data\") pod \"60189037-441a-4cb7-866c-6287bda16518\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") "
Oct 29 22:39:13.658812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.658796    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60189037-441a-4cb7-866c-6287bda16518-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60189037-441a-4cb7-866c-6287bda16518-istiod-ca-cert\") pod \"60189037-441a-4cb7-866c-6287bda16518\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") "
Oct 29 22:39:13.658857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.658845    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-certs\") pod \"60189037-441a-4cb7-866c-6287bda16518\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") "
Oct 29 22:39:13.658914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.658896    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-istio-token\" (UniqueName: \"kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-istio-token\") pod \"60189037-441a-4cb7-866c-6287bda16518\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") "
Oct 29 22:39:13.659088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.659070    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-envoy\") pod \"60189037-441a-4cb7-866c-6287bda16518\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") "
Oct 29 22:39:13.659198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.659180    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-kube-api-access-c79nd\" (UniqueName: \"kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-kube-api-access-c79nd\") pod \"60189037-441a-4cb7-866c-6287bda16518\" (UID: \"60189037-441a-4cb7-866c-6287bda16518\") "
Oct 29 22:39:13.662146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.659250    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60189037-441a-4cb7-866c-6287bda16518-istiod-ca-cert" pod "60189037-441a-4cb7-866c-6287bda16518" (UID: "60189037-441a-4cb7-866c-6287bda16518"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:13.662146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.659530    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-socket" pod "60189037-441a-4cb7-866c-6287bda16518" (UID: "60189037-441a-4cb7-866c-6287bda16518"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:13.662146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.659553    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-certs" pod "60189037-441a-4cb7-866c-6287bda16518" (UID: "60189037-441a-4cb7-866c-6287bda16518"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:13.662146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.659612    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-data" pod "60189037-441a-4cb7-866c-6287bda16518" (UID: "60189037-441a-4cb7-866c-6287bda16518"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:13.662146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.660914    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60189037-441a-4cb7-866c-6287bda16518-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:13.662146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.660939    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:13.662146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.660952    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:13.662146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.660963    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:13.662146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.661693    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-credential-socket" pod "60189037-441a-4cb7-866c-6287bda16518" (UID: "60189037-441a-4cb7-866c-6287bda16518"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:13.662715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.662614    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-istio-token" pod "60189037-441a-4cb7-866c-6287bda16518" (UID: "60189037-441a-4cb7-866c-6287bda16518"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:13.663103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.663079    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60189037-441a-4cb7-866c-6287bda16518-istio-podinfo" pod "60189037-441a-4cb7-866c-6287bda16518" (UID: "60189037-441a-4cb7-866c-6287bda16518"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:39:13.663263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.663221    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-envoy" pod "60189037-441a-4cb7-866c-6287bda16518" (UID: "60189037-441a-4cb7-866c-6287bda16518"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:13.663263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.663259    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-kube-api-access-c79nd" pod "60189037-441a-4cb7-866c-6287bda16518" (UID: "60189037-441a-4cb7-866c-6287bda16518"). InnerVolumeSpecName "kube-api-access-c79nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:13.781069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.780813    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:13.781069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.780877    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:13.781069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.780895    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c79nd\" (UniqueName: \"kubernetes.io/projected/60189037-441a-4cb7-866c-6287bda16518-kube-api-access-c79nd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:13.781069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.780910    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60189037-441a-4cb7-866c-6287bda16518-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:13.781069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.780926    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60189037-441a-4cb7-866c-6287bda16518-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:13.904700 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.904233    4860 patch_prober.go:29] interesting pod/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 29 22:39:13.904700 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:13.904297    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 29 22:39:14.334213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:14.334177    4860 patch_prober.go:29] interesting pod/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:39:14.334459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:14.334238    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 29 22:39:14.439730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:14.439696    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:14.653361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:14.653243    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:14.663616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:14.663575    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"
Oct 29 22:39:14.754473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:14.754394    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"]
Oct 29 22:39:14.759935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:14.759906    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-zbfz4"]
Oct 29 22:39:14.912354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:14.908964    4860 patch_prober.go:29] interesting pod/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 29 22:39:14.975913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:14.975866    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 29 22:39:15.259617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.259581    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:15.310350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.307543    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-xks-ca\") pod \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\" (UID: \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\") "
Oct 29 22:39:15.310350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.307642    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-openshift-ca\") pod \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\" (UID: \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\") "
Oct 29 22:39:15.310350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.307796    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e7ee26-39f3-4475-86ed-04a65e3e4f33-kube-api-access-94sjb\" (UniqueName: \"kubernetes.io/projected/18e7ee26-39f3-4475-86ed-04a65e3e4f33-kube-api-access-94sjb\") pod \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\" (UID: \"18e7ee26-39f3-4475-86ed-04a65e3e4f33\") "
Oct 29 22:39:15.314382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.311787    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-openshift-ca" pod "18e7ee26-39f3-4475-86ed-04a65e3e4f33" (UID: "18e7ee26-39f3-4475-86ed-04a65e3e4f33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:15.314382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.313267    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e7ee26-39f3-4475-86ed-04a65e3e4f33-kube-api-access-94sjb" pod "18e7ee26-39f3-4475-86ed-04a65e3e4f33" (UID: "18e7ee26-39f3-4475-86ed-04a65e3e4f33"). InnerVolumeSpecName "kube-api-access-94sjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:15.326588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.324377    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-xks-ca" pod "18e7ee26-39f3-4475-86ed-04a65e3e4f33" (UID: "18e7ee26-39f3-4475-86ed-04a65e3e4f33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:39:15.337496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.334871    4860 patch_prober.go:29] interesting pod/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:39:15.337496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.334951    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 29 22:39:15.411178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.411134    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94sjb\" (UniqueName: \"kubernetes.io/projected/18e7ee26-39f3-4475-86ed-04a65e3e4f33-kube-api-access-94sjb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:15.411178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.411181    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:15.411528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.411198    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e7ee26-39f3-4475-86ed-04a65e3e4f33-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:15.483902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.480676    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:15.675977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.675901    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"
Oct 29 22:39:15.675977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.675945    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn" event={"ID":"18e7ee26-39f3-4475-86ed-04a65e3e4f33","Type":"ContainerDied","Data":"82365b1767ba1712f54d8621399eaadf6db7f6ce57690d733e4b22de21891cdc"}
Oct 29 22:39:15.676164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.675996    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82365b1767ba1712f54d8621399eaadf6db7f6ce57690d733e4b22de21891cdc"
Oct 29 22:39:15.895509 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.895479    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:39:15.895911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.895893    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:39:15.896641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:15.896619    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:39:16.114625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:16.114590    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60189037-441a-4cb7-866c-6287bda16518" path="/var/lib/kubelet/pods/60189037-441a-4cb7-866c-6287bda16518/volumes"
Oct 29 22:39:16.339292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:16.339263    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:39:16.339787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:16.339770    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:39:16.340567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:16.340550    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:39:17.171154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:17.171128    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:18.056099 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:39:18.056058    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:39:18.365275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:18.365207    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:39:18.368203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:18.368182    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:39:18.732986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:18.732914    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:19.485196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:19.485163    4860 scope.go:122] "RemoveContainer" containerID="165f414f65fb13c90d6353ac338c84ebe4de3709ba8d9a66a425787fd4516d41"
Oct 29 22:39:19.496865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:19.496845    4860 scope.go:122] "RemoveContainer" containerID="fd149bfcdc3d624f9335a375e233919b96d7a65b924156d2320a7914be8db78b"
Oct 29 22:39:19.507420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:19.507400    4860 scope.go:122] "RemoveContainer" containerID="8f6151214a5bc27d8cd48d00bc6551293620e3e66c87f963f7a9a98eea0e3b4a"
Oct 29 22:39:19.517519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:19.517502    4860 scope.go:122] "RemoveContainer" containerID="de5084434985ce3fdcb924d549ffb3c6e7c787035045a92e356a9120c2aade58"
Oct 29 22:39:19.531278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:19.531254    4860 scope.go:122] "RemoveContainer" containerID="ea369d1ba889635e8625129760a7ba1a10a6d53d9ab447c8a87071f4fa454653"
Oct 29 22:39:19.541141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:19.541121    4860 scope.go:122] "RemoveContainer" containerID="510d62aaca6d91378c892bd5a58ace1bf9cdc9fd46149e5bded1285fbf821526"
Oct 29 22:39:19.550515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:19.550497    4860 scope.go:122] "RemoveContainer" containerID="7c9c6c0976e363cb89a2b918b75b6fe729c10cd824955bfe401d221c037061bc"
Oct 29 22:39:19.560034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:19.560020    4860 scope.go:122] "RemoveContainer" containerID="608d511a213d0ee5a64e1b2dcdd53cc413d02f5118022d82cd9b90141e48b12a"
Oct 29 22:39:20.284336 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:20.284293    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:21.540114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:21.540083    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"]
Oct 29 22:39:21.674809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:21.674780    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:21.804276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:21.804188    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:22.104142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:22.104037    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:23.033656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:23.033617    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:23.768447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:23.768410    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:25.500652 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:25.500618    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:26.540845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:26.540807    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469" podUID="a61010f1-3be5-4385-bf1d-3cb8331b6058" containerName="payload-pre-processing" containerID="cri-o://9eb112ab7c1a39e891421c9025b1fba29b20fee55e2b8a70902bcb4855c2858c" gracePeriod=25
Oct 29 22:39:26.788655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:26.788616    4860 generic.go:356] "Generic (PLEG): container finished" podID="a61010f1-3be5-4385-bf1d-3cb8331b6058" containerID="9eb112ab7c1a39e891421c9025b1fba29b20fee55e2b8a70902bcb4855c2858c" exitCode=0
Oct 29 22:39:26.788800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:26.788667    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469" event={"ID":"a61010f1-3be5-4385-bf1d-3cb8331b6058","Type":"ContainerDied","Data":"9eb112ab7c1a39e891421c9025b1fba29b20fee55e2b8a70902bcb4855c2858c"}
Oct 29 22:39:26.850455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:26.850432    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"
Oct 29 22:39:26.961959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:26.961927    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61010f1-3be5-4385-bf1d-3cb8331b6058-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61010f1-3be5-4385-bf1d-3cb8331b6058-plugins-config-volume\") pod \"a61010f1-3be5-4385-bf1d-3cb8331b6058\" (UID: \"a61010f1-3be5-4385-bf1d-3cb8331b6058\") "
Oct 29 22:39:26.962116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:26.962029    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61010f1-3be5-4385-bf1d-3cb8331b6058-kube-api-access-qnxxr\" (UniqueName: \"kubernetes.io/projected/a61010f1-3be5-4385-bf1d-3cb8331b6058-kube-api-access-qnxxr\") pod \"a61010f1-3be5-4385-bf1d-3cb8331b6058\" (UID: \"a61010f1-3be5-4385-bf1d-3cb8331b6058\") "
Oct 29 22:39:26.962310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:26.962284    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61010f1-3be5-4385-bf1d-3cb8331b6058-plugins-config-volume" pod "a61010f1-3be5-4385-bf1d-3cb8331b6058" (UID: "a61010f1-3be5-4385-bf1d-3cb8331b6058"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:26.962750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:26.962733    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61010f1-3be5-4385-bf1d-3cb8331b6058-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:26.964195 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:26.964175    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61010f1-3be5-4385-bf1d-3cb8331b6058-kube-api-access-qnxxr" pod "a61010f1-3be5-4385-bf1d-3cb8331b6058" (UID: "a61010f1-3be5-4385-bf1d-3cb8331b6058"). InnerVolumeSpecName "kube-api-access-qnxxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:27.038738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:27.038715    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:27.090660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:27.078571    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnxxr\" (UniqueName: \"kubernetes.io/projected/a61010f1-3be5-4385-bf1d-3cb8331b6058-kube-api-access-qnxxr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:27.807286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:27.807249    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"
Oct 29 22:39:27.807286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:27.807271    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469" event={"ID":"a61010f1-3be5-4385-bf1d-3cb8331b6058","Type":"ContainerDied","Data":"6cdc8ee215f87f1eb0b060f89e1f09f73573ad930c60c1111ed7019e2bbd4906"}
Oct 29 22:39:27.807793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:27.807379    4860 scope.go:122] "RemoveContainer" containerID="9eb112ab7c1a39e891421c9025b1fba29b20fee55e2b8a70902bcb4855c2858c"
Oct 29 22:39:27.867422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:27.867396    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"]
Oct 29 22:39:27.879468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:27.879444    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee886f-5979bf8887-hr469"]
Oct 29 22:39:28.115427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:28.115350    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a61010f1-3be5-4385-bf1d-3cb8331b6058" path="/var/lib/kubelet/pods/a61010f1-3be5-4385-bf1d-3cb8331b6058/volumes"
Oct 29 22:39:28.722103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:28.722066    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:30.655334 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:30.655301    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:32.259675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:32.259642    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:32.799149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:32.799111    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:33.241086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:33.241051    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:33.717364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:33.717285    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:33.835084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:33.835056    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:33.953347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:33.953016    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:34.172493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:34.172421    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:34.303585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:34.303551    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:35.618515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:35.618481    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:36.372765 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:36.372730    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:36.545679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:36.545645    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:36.694502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:36.694468    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"]
Oct 29 22:39:36.694904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:36.694697    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4" podUID="88686c92-66f4-4846-8c9e-bf969f13efd4" containerName="istio-proxy" containerID="cri-o://f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79" gracePeriod=30
Oct 29 22:39:36.709531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:36.709497    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"]
Oct 29 22:39:36.709766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:36.709739    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5" podUID="72edf772-cda9-4f90-8725-4913072cc67c" containerName="istio-proxy" containerID="cri-o://739765837b1a023c885f79404d14deab999fa12895edcba4008591377b55eeb5" gracePeriod=30
Oct 29 22:39:37.366130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:37.366100    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:37.954175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:37.954144    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:38.100570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:38.100531    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:39.290091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.290048    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:39.671225 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.671140    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:39.750822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.750783    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"]
Oct 29 22:39:39.751888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.751865    4860 cpu_manager.go:414] "containerMap: removing container" podUID="60189037-441a-4cb7-866c-6287bda16518" containerName="istio-proxy"
Oct 29 22:39:39.751888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.751885    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60189037-441a-4cb7-866c-6287bda16518" containerName="istio-proxy"
Oct 29 22:39:39.752056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.751893    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a61010f1-3be5-4385-bf1d-3cb8331b6058" containerName="payload-pre-processing"
Oct 29 22:39:39.752056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.751899    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61010f1-3be5-4385-bf1d-3cb8331b6058" containerName="payload-pre-processing"
Oct 29 22:39:39.752056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.751916    4860 cpu_manager.go:414] "containerMap: removing container" podUID="18e7ee26-39f3-4475-86ed-04a65e3e4f33" containerName="revoke-keys"
Oct 29 22:39:39.752056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.751921    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e7ee26-39f3-4475-86ed-04a65e3e4f33" containerName="revoke-keys"
Oct 29 22:39:39.752210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.752079    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="60189037-441a-4cb7-866c-6287bda16518" containerName="istio-proxy"
Oct 29 22:39:39.752210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.752104    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61010f1-3be5-4385-bf1d-3cb8331b6058" containerName="payload-pre-processing"
Oct 29 22:39:39.752210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.752114    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e7ee26-39f3-4475-86ed-04a65e3e4f33" containerName="revoke-keys"
Oct 29 22:39:39.756091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.756073    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"
Oct 29 22:39:39.773405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.773365    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"]
Oct 29 22:39:39.781340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.781308    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c296a3-dockercfg-7ckpn\""
Oct 29 22:39:39.781460 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.781370    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c296a3\""
Oct 29 22:39:39.803705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.803675    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:39.964018 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:39.963982    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg8sd\" (UniqueName: \"kubernetes.io/projected/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-kube-api-access-gg8sd\") pod \"payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68\" (UID: \"e40e4a62-bd4d-4df7-a185-efcaf2d1071f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"
Oct 29 22:39:40.011674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:40.011634    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68\" (UID: \"e40e4a62-bd4d-4df7-a185-efcaf2d1071f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"
Oct 29 22:39:40.113763 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:40.113730    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg8sd\" (UniqueName: \"kubernetes.io/projected/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-kube-api-access-gg8sd\") pod \"payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68\" (UID: \"e40e4a62-bd4d-4df7-a185-efcaf2d1071f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"
Oct 29 22:39:40.114266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:40.114230    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68\" (UID: \"e40e4a62-bd4d-4df7-a185-efcaf2d1071f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"
Oct 29 22:39:40.115459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:40.115431    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68\" (UID: \"e40e4a62-bd4d-4df7-a185-efcaf2d1071f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"
Oct 29 22:39:40.135022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:40.134993    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg8sd\" (UniqueName: \"kubernetes.io/projected/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-kube-api-access-gg8sd\") pod \"payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68\" (UID: \"e40e4a62-bd4d-4df7-a185-efcaf2d1071f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"
Oct 29 22:39:40.371081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:40.370997    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"
Oct 29 22:39:40.676583 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:39:40.676550    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40e4a62_bd4d_4df7_a185_efcaf2d1071f.slice/crio-4169468c399e464ae9b6b05836470b3722b6045f77577d79169da34ee216ccbe WatchSource:0}: Error finding container 4169468c399e464ae9b6b05836470b3722b6045f77577d79169da34ee216ccbe: Status 404 returned error can't find the container with id 4169468c399e464ae9b6b05836470b3722b6045f77577d79169da34ee216ccbe
Oct 29 22:39:40.681271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:40.681246    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"]
Oct 29 22:39:40.973049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:40.973013    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:41.045142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:41.045103    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68" event={"ID":"e40e4a62-bd4d-4df7-a185-efcaf2d1071f","Type":"ContainerStarted","Data":"eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554"}
Oct 29 22:39:41.045409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:41.045164    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68" event={"ID":"e40e4a62-bd4d-4df7-a185-efcaf2d1071f","Type":"ContainerStarted","Data":"4169468c399e464ae9b6b05836470b3722b6045f77577d79169da34ee216ccbe"}
Oct 29 22:39:41.045409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:41.045192    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"
Oct 29 22:39:41.093095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:41.093041    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68" podStartSLOduration=2.093027737 podStartE2EDuration="2.093027737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:41.08683659 +0000 UTC m=+1523.111612720" watchObservedRunningTime="2026-10-29 22:39:41.093027737 +0000 UTC m=+1523.117803892"
Oct 29 22:39:42.017096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.017065    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:39:42.056041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.056012    4860 generic.go:356] "Generic (PLEG): container finished" podID="72edf772-cda9-4f90-8725-4913072cc67c" containerID="739765837b1a023c885f79404d14deab999fa12895edcba4008591377b55eeb5" exitCode=0
Oct 29 22:39:42.056197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.056082    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5" event={"ID":"72edf772-cda9-4f90-8725-4913072cc67c","Type":"ContainerDied","Data":"739765837b1a023c885f79404d14deab999fa12895edcba4008591377b55eeb5"}
Oct 29 22:39:42.057799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.057779    4860 generic.go:356] "Generic (PLEG): container finished" podID="88686c92-66f4-4846-8c9e-bf969f13efd4" containerID="f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79" exitCode=0
Oct 29 22:39:42.057895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.057838    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"
Oct 29 22:39:42.057963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.057908    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4" event={"ID":"88686c92-66f4-4846-8c9e-bf969f13efd4","Type":"ContainerDied","Data":"f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79"}
Oct 29 22:39:42.058014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.057963    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4" event={"ID":"88686c92-66f4-4846-8c9e-bf969f13efd4","Type":"ContainerDied","Data":"f390eeb3c49a826eb3f63ac4a5d1a2e4aed0dfa2bfce233bba90f3ea021bec66"}
Oct 29 22:39:42.058014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.057978    4860 scope.go:122] "RemoveContainer" containerID="f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79"
Oct 29 22:39:42.076655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.076060    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-envoy\") pod \"88686c92-66f4-4846-8c9e-bf969f13efd4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") "
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.077786    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-kube-api-access-qdxtm\" (UniqueName: \"kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-kube-api-access-qdxtm\") pod \"88686c92-66f4-4846-8c9e-bf969f13efd4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") "
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.077911    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-socket\") pod \"88686c92-66f4-4846-8c9e-bf969f13efd4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") "
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.078016    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88686c92-66f4-4846-8c9e-bf969f13efd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88686c92-66f4-4846-8c9e-bf969f13efd4-istiod-ca-cert\") pod \"88686c92-66f4-4846-8c9e-bf969f13efd4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") "
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.078097    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-data\") pod \"88686c92-66f4-4846-8c9e-bf969f13efd4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") "
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.078209    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-token\" (UniqueName: \"kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-token\") pod \"88686c92-66f4-4846-8c9e-bf969f13efd4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") "
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.078369    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-credential-socket\") pod \"88686c92-66f4-4846-8c9e-bf969f13efd4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") "
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.078444    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-podinfo\") pod \"88686c92-66f4-4846-8c9e-bf969f13efd4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") "
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.078537    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-certs\") pod \"88686c92-66f4-4846-8c9e-bf969f13efd4\" (UID: \"88686c92-66f4-4846-8c9e-bf969f13efd4\") "
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.080147    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-certs" pod "88686c92-66f4-4846-8c9e-bf969f13efd4" (UID: "88686c92-66f4-4846-8c9e-bf969f13efd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.080371    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-data" pod "88686c92-66f4-4846-8c9e-bf969f13efd4" (UID: "88686c92-66f4-4846-8c9e-bf969f13efd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.081156    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-socket" pod "88686c92-66f4-4846-8c9e-bf969f13efd4" (UID: "88686c92-66f4-4846-8c9e-bf969f13efd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.081650    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88686c92-66f4-4846-8c9e-bf969f13efd4-istiod-ca-cert" pod "88686c92-66f4-4846-8c9e-bf969f13efd4" (UID: "88686c92-66f4-4846-8c9e-bf969f13efd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.081722    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-envoy" pod "88686c92-66f4-4846-8c9e-bf969f13efd4" (UID: "88686c92-66f4-4846-8c9e-bf969f13efd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:42.082421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.081961    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-credential-socket" pod "88686c92-66f4-4846-8c9e-bf969f13efd4" (UID: "88686c92-66f4-4846-8c9e-bf969f13efd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:42.083390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.083369    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-token" pod "88686c92-66f4-4846-8c9e-bf969f13efd4" (UID: "88686c92-66f4-4846-8c9e-bf969f13efd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:42.087917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.087887    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-podinfo" pod "88686c92-66f4-4846-8c9e-bf969f13efd4" (UID: "88686c92-66f4-4846-8c9e-bf969f13efd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:39:42.089574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.089548    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-kube-api-access-qdxtm" pod "88686c92-66f4-4846-8c9e-bf969f13efd4" (UID: "88686c92-66f4-4846-8c9e-bf969f13efd4"). InnerVolumeSpecName "kube-api-access-qdxtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:42.107261 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.107231    4860 scope.go:122] "RemoveContainer" containerID="f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79"
Oct 29 22:39:42.108497 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:39:42.108467    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79\": container with ID starting with f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79 not found: ID does not exist" containerID="f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79"
Oct 29 22:39:42.108605 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.108509    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79"} err="failed to get container status \"f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79\": rpc error: code = NotFound desc = could not find container \"f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79\": container with ID starting with f9e3fc702d0661b00e0e1ce1697a94f87d7afef68293a2cc547900d57a4cfb79 not found: ID does not exist"
Oct 29 22:39:42.114463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.114439    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:39:42.120525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.120504    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:42.180705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.180063    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-istio-token\" (UniqueName: \"kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-istio-token\") pod \"72edf772-cda9-4f90-8725-4913072cc67c\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") "
Oct 29 22:39:42.180705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.180150    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-data\") pod \"72edf772-cda9-4f90-8725-4913072cc67c\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") "
Oct 29 22:39:42.180705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.180263    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-certs\") pod \"72edf772-cda9-4f90-8725-4913072cc67c\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") "
Oct 29 22:39:42.180705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.180282    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72edf772-cda9-4f90-8725-4913072cc67c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72edf772-cda9-4f90-8725-4913072cc67c-istio-podinfo\") pod \"72edf772-cda9-4f90-8725-4913072cc67c\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") "
Oct 29 22:39:42.180705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.180306    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72edf772-cda9-4f90-8725-4913072cc67c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72edf772-cda9-4f90-8725-4913072cc67c-istiod-ca-cert\") pod \"72edf772-cda9-4f90-8725-4913072cc67c\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") "
Oct 29 22:39:42.180705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.180402    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-socket\") pod \"72edf772-cda9-4f90-8725-4913072cc67c\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") "
Oct 29 22:39:42.180705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.180447    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-envoy\") pod \"72edf772-cda9-4f90-8725-4913072cc67c\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") "
Oct 29 22:39:42.180705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.180512    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-kube-api-access-n98p5\" (UniqueName: \"kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-kube-api-access-n98p5\") pod \"72edf772-cda9-4f90-8725-4913072cc67c\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") "
Oct 29 22:39:42.180705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.180574    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-credential-socket\") pod \"72edf772-cda9-4f90-8725-4913072cc67c\" (UID: \"72edf772-cda9-4f90-8725-4913072cc67c\") "
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.181421    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.181444    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.181460    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.181473    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.181486    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdxtm\" (UniqueName: \"kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-kube-api-access-qdxtm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.181500    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.181514    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88686c92-66f4-4846-8c9e-bf969f13efd4-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.181527    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.181539    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88686c92-66f4-4846-8c9e-bf969f13efd4-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.184096    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72edf772-cda9-4f90-8725-4913072cc67c-istiod-ca-cert" pod "72edf772-cda9-4f90-8725-4913072cc67c" (UID: "72edf772-cda9-4f90-8725-4913072cc67c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.184241    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-socket" pod "72edf772-cda9-4f90-8725-4913072cc67c" (UID: "72edf772-cda9-4f90-8725-4913072cc67c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.184391    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-data" pod "72edf772-cda9-4f90-8725-4913072cc67c" (UID: "72edf772-cda9-4f90-8725-4913072cc67c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:42.184886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.184573    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-certs" pod "72edf772-cda9-4f90-8725-4913072cc67c" (UID: "72edf772-cda9-4f90-8725-4913072cc67c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:42.186047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.186016    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-credential-socket" pod "72edf772-cda9-4f90-8725-4913072cc67c" (UID: "72edf772-cda9-4f90-8725-4913072cc67c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:42.186905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.186876    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-kube-api-access-n98p5" pod "72edf772-cda9-4f90-8725-4913072cc67c" (UID: "72edf772-cda9-4f90-8725-4913072cc67c"). InnerVolumeSpecName "kube-api-access-n98p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:42.187310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.187276    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72edf772-cda9-4f90-8725-4913072cc67c-istio-podinfo" pod "72edf772-cda9-4f90-8725-4913072cc67c" (UID: "72edf772-cda9-4f90-8725-4913072cc67c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:39:42.188645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.188622    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-envoy" pod "72edf772-cda9-4f90-8725-4913072cc67c" (UID: "72edf772-cda9-4f90-8725-4913072cc67c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:39:42.193006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.192984    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-istio-token" pod "72edf772-cda9-4f90-8725-4913072cc67c" (UID: "72edf772-cda9-4f90-8725-4913072cc67c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:42.283427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.283350    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n98p5\" (UniqueName: \"kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-kube-api-access-n98p5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.283427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.283380    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.283427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.283390    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72edf772-cda9-4f90-8725-4913072cc67c-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.283427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.283398    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.283427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.283407    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.283427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.283414    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72edf772-cda9-4f90-8725-4913072cc67c-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.283427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.283421    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72edf772-cda9-4f90-8725-4913072cc67c-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.283427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.283429    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.283920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.283436    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72edf772-cda9-4f90-8725-4913072cc67c-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:42.403916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.403883    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"]
Oct 29 22:39:42.408891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.408865    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-nkds4"]
Oct 29 22:39:42.780550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:42.780521    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:43.069089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:43.069003    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"
Oct 29 22:39:43.069089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:43.069039    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5" event={"ID":"72edf772-cda9-4f90-8725-4913072cc67c","Type":"ContainerDied","Data":"36b7de2eb08aa7e0798e6db7e0d43b71854f5b6cd9e8b7ba81ee9614cf8bb6bd"}
Oct 29 22:39:43.069670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:43.069110    4860 scope.go:122] "RemoveContainer" containerID="739765837b1a023c885f79404d14deab999fa12895edcba4008591377b55eeb5"
Oct 29 22:39:43.128546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:43.128522    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"]
Oct 29 22:39:43.142549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:43.142522    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-4h4z5"]
Oct 29 22:39:44.113888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:44.113855    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72edf772-cda9-4f90-8725-4913072cc67c" path="/var/lib/kubelet/pods/72edf772-cda9-4f90-8725-4913072cc67c/volumes"
Oct 29 22:39:44.114359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:44.114340    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88686c92-66f4-4846-8c9e-bf969f13efd4" path="/var/lib/kubelet/pods/88686c92-66f4-4846-8c9e-bf969f13efd4/volumes"
Oct 29 22:39:44.830068 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:44.830037    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:45.518039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:45.518001    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:45.962674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:45.962635    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:46.578712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.578661    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"]
Oct 29 22:39:46.580001 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.579975    4860 cpu_manager.go:414] "containerMap: removing container" podUID="72edf772-cda9-4f90-8725-4913072cc67c" containerName="istio-proxy"
Oct 29 22:39:46.580097 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.580005    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72edf772-cda9-4f90-8725-4913072cc67c" containerName="istio-proxy"
Oct 29 22:39:46.580097 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.580071    4860 cpu_manager.go:414] "containerMap: removing container" podUID="88686c92-66f4-4846-8c9e-bf969f13efd4" containerName="istio-proxy"
Oct 29 22:39:46.580097 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.580079    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88686c92-66f4-4846-8c9e-bf969f13efd4" containerName="istio-proxy"
Oct 29 22:39:46.580396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.580376    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="72edf772-cda9-4f90-8725-4913072cc67c" containerName="istio-proxy"
Oct 29 22:39:46.580494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.580416    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="88686c92-66f4-4846-8c9e-bf969f13efd4" containerName="istio-proxy"
Oct 29 22:39:46.587172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.587150    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.604861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.604834    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"]
Oct 29 22:39:46.611575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.611551    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8d985a-openshift-default-dockercfg-d87jf\""
Oct 29 22:39:46.668843 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.668804    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:46.690433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.690384    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-credential-socket\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.690594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.690438    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-podinfo\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.690594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.690544    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-envoy\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.690689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.690611    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-certs\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.690753 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.690729    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtn4k\" (UniqueName: \"kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-kube-api-access-rtn4k\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.690954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.690935    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-socket\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.691008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.690969    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afafb7-388b-4114-82a8-adacbbb15f5d-istiod-ca-cert\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.691153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.691135    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-data\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.691188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.691167    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-token\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.801735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.800447    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-certs\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.801735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.800532    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtn4k\" (UniqueName: \"kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-kube-api-access-rtn4k\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.801735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.800704    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-socket\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.801735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.800732    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afafb7-388b-4114-82a8-adacbbb15f5d-istiod-ca-cert\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.801735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.800847    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-data\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.801735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.800880    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-token\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.801735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.800980    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-credential-socket\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.801735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.801020    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-podinfo\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.801735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.801059    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-envoy\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.803089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.803059    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afafb7-388b-4114-82a8-adacbbb15f5d-istiod-ca-cert\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.808390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.805656    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-certs\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.809226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.809198    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-socket\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.810002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.809959    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-data\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.810426 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.810402    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-credential-socket\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.815553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.815389    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-podinfo\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.825634 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.825614    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-envoy\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.838544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.838483    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtn4k\" (UniqueName: \"kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-kube-api-access-rtn4k\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.840067 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.840039    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-token\") pod \"e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") " pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:46.910862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:46.910829    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:47.137611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:47.137304    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:47.265308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:47.265280    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:47.329028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:47.328953    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"]
Oct 29 22:39:47.330084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:47.330037    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:39:47.330195 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:47.330120    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:39:47.330273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:47.330198    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:39:47.557344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:47.556745    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:47.692354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:47.692302    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:48.059070 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:39:48.058990    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:39:48.250444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:48.250411    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" event={"ID":"a3afafb7-388b-4114-82a8-adacbbb15f5d","Type":"ContainerStarted","Data":"de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b"}
Oct 29 22:39:48.250651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:48.250473    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" event={"ID":"a3afafb7-388b-4114-82a8-adacbbb15f5d","Type":"ContainerStarted","Data":"a2db88d13d722ff69017f39927e45d64b9e1fd981dbc5adec86d84f86d975913"}
Oct 29 22:39:48.325714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:48.325623    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" podStartSLOduration=2.32560995 podStartE2EDuration="2.32560995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:48.310304119 +0000 UTC m=+1530.335080250" watchObservedRunningTime="2026-10-29 22:39:48.32560995 +0000 UTC m=+1530.350386139"
Oct 29 22:39:48.911876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:48.911841    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:48.917662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:48.917633    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:49.064917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:49.064886    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:49.259004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:49.258974    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:49.260089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:49.260071    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:39:50.702358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:50.702306    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:50.838357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:50.837305    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:52.061956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.061927    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"
Oct 29 22:39:52.448584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.448511    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:52.481628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.480496    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"]
Oct 29 22:39:52.487246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.487219    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:52.512338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.510729    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:39:52.512338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.511402    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:39:52.517863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.517840    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"]
Oct 29 22:39:52.552282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.552245    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk\" (UID: \"d95156da-7686-41d2-b02f-54f07fac9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:52.552484 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.552429    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk\" (UID: \"d95156da-7686-41d2-b02f-54f07fac9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:52.552716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.552691    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r29z9\" (UniqueName: \"kubernetes.io/projected/d95156da-7686-41d2-b02f-54f07fac9d0d-kube-api-access-r29z9\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk\" (UID: \"d95156da-7686-41d2-b02f-54f07fac9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:52.596942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.596909    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"]
Oct 29 22:39:52.597459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.597431    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:52.603522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.603502    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"
Oct 29 22:39:52.622283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.622258    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"]
Oct 29 22:39:52.635176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.630754    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8d985a-dockercfg-q44wj\""
Oct 29 22:39:52.654215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.654177    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk\" (UID: \"d95156da-7686-41d2-b02f-54f07fac9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:52.655044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.654298    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk\" (UID: \"d95156da-7686-41d2-b02f-54f07fac9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:52.655044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.654369    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzhf\" (UniqueName: \"kubernetes.io/projected/369bdcac-0746-4921-8a95-1a3006e4bc04-kube-api-access-bmzhf\") pod \"payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4\" (UID: \"369bdcac-0746-4921-8a95-1a3006e4bc04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"
Oct 29 22:39:52.655652 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.655615    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r29z9\" (UniqueName: \"kubernetes.io/projected/d95156da-7686-41d2-b02f-54f07fac9d0d-kube-api-access-r29z9\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk\" (UID: \"d95156da-7686-41d2-b02f-54f07fac9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:52.655811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.655669    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369bdcac-0746-4921-8a95-1a3006e4bc04-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4\" (UID: \"369bdcac-0746-4921-8a95-1a3006e4bc04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"
Oct 29 22:39:52.656847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.656820    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk\" (UID: \"d95156da-7686-41d2-b02f-54f07fac9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:52.658664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.658634    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk\" (UID: \"d95156da-7686-41d2-b02f-54f07fac9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:52.682773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.682745    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r29z9\" (UniqueName: \"kubernetes.io/projected/d95156da-7686-41d2-b02f-54f07fac9d0d-kube-api-access-r29z9\") pod \"maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk\" (UID: \"d95156da-7686-41d2-b02f-54f07fac9d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:52.696731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.696707    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"]
Oct 29 22:39:52.699533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.699464    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8d985a\""
Oct 29 22:39:52.702224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.702204    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"
Oct 29 22:39:52.750353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.749709    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"]
Oct 29 22:39:52.784182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.782113    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s\" (UID: \"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"
Oct 29 22:39:52.784182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.782177    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369bdcac-0746-4921-8a95-1a3006e4bc04-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4\" (UID: \"369bdcac-0746-4921-8a95-1a3006e4bc04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"
Oct 29 22:39:52.784182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.782503    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzhf\" (UniqueName: \"kubernetes.io/projected/369bdcac-0746-4921-8a95-1a3006e4bc04-kube-api-access-bmzhf\") pod \"payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4\" (UID: \"369bdcac-0746-4921-8a95-1a3006e4bc04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"
Oct 29 22:39:52.784182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.782568    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqq8\" (UniqueName: \"kubernetes.io/projected/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-kube-api-access-7gqq8\") pod \"payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s\" (UID: \"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"
Oct 29 22:39:52.784182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.783737    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369bdcac-0746-4921-8a95-1a3006e4bc04-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4\" (UID: \"369bdcac-0746-4921-8a95-1a3006e4bc04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"
Oct 29 22:39:52.871373 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.869636    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:52.995915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.995825    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqq8\" (UniqueName: \"kubernetes.io/projected/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-kube-api-access-7gqq8\") pod \"payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s\" (UID: \"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"
Oct 29 22:39:52.996381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.996354    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s\" (UID: \"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"
Oct 29 22:39:52.997960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:52.997936    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s\" (UID: \"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"
Oct 29 22:39:53.169585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:53.161112    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzhf\" (UniqueName: \"kubernetes.io/projected/369bdcac-0746-4921-8a95-1a3006e4bc04-kube-api-access-bmzhf\") pod \"payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4\" (UID: \"369bdcac-0746-4921-8a95-1a3006e4bc04\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"
Oct 29 22:39:53.180055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:53.180024    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqq8\" (UniqueName: \"kubernetes.io/projected/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-kube-api-access-7gqq8\") pod \"payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s\" (UID: \"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"
Oct 29 22:39:53.222489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:53.219813    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"
Oct 29 22:39:53.343116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:53.343080    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"
Oct 29 22:39:53.494473 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:39:53.494443    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95156da_7686_41d2_b02f_54f07fac9d0d.slice/crio-8c7cc4443dbaeed88c1eac080fa9737a7040897bafc641f4eb5195b4455f74d2 WatchSource:0}: Error finding container 8c7cc4443dbaeed88c1eac080fa9737a7040897bafc641f4eb5195b4455f74d2: Status 404 returned error can't find the container with id 8c7cc4443dbaeed88c1eac080fa9737a7040897bafc641f4eb5195b4455f74d2
Oct 29 22:39:53.503288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:53.501417    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"]
Oct 29 22:39:53.541386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:53.539816    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:53.637366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:53.637335    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"]
Oct 29 22:39:53.670669 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:39:53.670633    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369bdcac_0746_4921_8a95_1a3006e4bc04.slice/crio-1e443245a4b7b2ad703966976676e5acf6e49a615ba9cb43d293d346606f795f WatchSource:0}: Error finding container 1e443245a4b7b2ad703966976676e5acf6e49a615ba9cb43d293d346606f795f: Status 404 returned error can't find the container with id 1e443245a4b7b2ad703966976676e5acf6e49a615ba9cb43d293d346606f795f
Oct 29 22:39:53.671202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:53.671166    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:53.679046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:53.679028    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"]
Oct 29 22:39:53.684277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:53.684252    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cae55b-bvvkw"]
Oct 29 22:39:53.717010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:53.716960    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"]
Oct 29 22:39:54.114146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.114070    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0ffc8b5-da6b-4ced-860e-e87812cda589" path="/var/lib/kubelet/pods/a0ffc8b5-da6b-4ced-860e-e87812cda589/volumes"
Oct 29 22:39:54.161476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.161423    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:54.381550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.381467    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4" event={"ID":"369bdcac-0746-4921-8a95-1a3006e4bc04","Type":"ContainerStarted","Data":"18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9"}
Oct 29 22:39:54.381550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.381531    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4" event={"ID":"369bdcac-0746-4921-8a95-1a3006e4bc04","Type":"ContainerStarted","Data":"1e443245a4b7b2ad703966976676e5acf6e49a615ba9cb43d293d346606f795f"}
Oct 29 22:39:54.381970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.381562    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"
Oct 29 22:39:54.383369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.383314    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s" event={"ID":"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb","Type":"ContainerStarted","Data":"7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81"}
Oct 29 22:39:54.383513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.383380    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s" event={"ID":"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb","Type":"ContainerStarted","Data":"7121419f3fdfbe21f5241c3226c3bd2779a4624254680f492fbf1e562468cd58"}
Oct 29 22:39:54.383513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.383467    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"
Oct 29 22:39:54.384955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.384932    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk" event={"ID":"d95156da-7686-41d2-b02f-54f07fac9d0d","Type":"ContainerStarted","Data":"3053d7411a3ca9ee2c536874119a14e875e2ddd6958037fb0853049743370768"}
Oct 29 22:39:54.385025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.384979    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk" event={"ID":"d95156da-7686-41d2-b02f-54f07fac9d0d","Type":"ContainerStarted","Data":"8c7cc4443dbaeed88c1eac080fa9737a7040897bafc641f4eb5195b4455f74d2"}
Oct 29 22:39:54.428896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.428854    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4" podStartSLOduration=2.428843692 podStartE2EDuration="2.428843692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:54.423565644 +0000 UTC m=+1536.448341775" watchObservedRunningTime="2026-10-29 22:39:54.428843692 +0000 UTC m=+1536.453619823"
Oct 29 22:39:54.468806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.468769    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk" podStartSLOduration=2.468755943 podStartE2EDuration="2.468755943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:54.464395449 +0000 UTC m=+1536.489171580" watchObservedRunningTime="2026-10-29 22:39:54.468755943 +0000 UTC m=+1536.493532073"
Oct 29 22:39:54.510927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:54.510877    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s" podStartSLOduration=2.510863492 podStartE2EDuration="2.510863492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:54.508114526 +0000 UTC m=+1536.532890657" watchObservedRunningTime="2026-10-29 22:39:54.510863492 +0000 UTC m=+1536.535639665"
Oct 29 22:39:55.395620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:55.395586    4860 generic.go:356] "Generic (PLEG): container finished" podID="d95156da-7686-41d2-b02f-54f07fac9d0d" containerID="3053d7411a3ca9ee2c536874119a14e875e2ddd6958037fb0853049743370768" exitCode=0
Oct 29 22:39:55.396035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:55.395676    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk" event={"ID":"d95156da-7686-41d2-b02f-54f07fac9d0d","Type":"ContainerDied","Data":"3053d7411a3ca9ee2c536874119a14e875e2ddd6958037fb0853049743370768"}
Oct 29 22:39:55.442168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:55.442137    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:56.568795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:56.568768    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:56.653647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:56.653609    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95156da-7686-41d2-b02f-54f07fac9d0d-kube-api-access-r29z9\" (UniqueName: \"kubernetes.io/projected/d95156da-7686-41d2-b02f-54f07fac9d0d-kube-api-access-r29z9\") pod \"d95156da-7686-41d2-b02f-54f07fac9d0d\" (UID: \"d95156da-7686-41d2-b02f-54f07fac9d0d\") "
Oct 29 22:39:56.653805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:56.653707    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-openshift-ca\") pod \"d95156da-7686-41d2-b02f-54f07fac9d0d\" (UID: \"d95156da-7686-41d2-b02f-54f07fac9d0d\") "
Oct 29 22:39:56.653878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:56.653860    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-xks-ca\") pod \"d95156da-7686-41d2-b02f-54f07fac9d0d\" (UID: \"d95156da-7686-41d2-b02f-54f07fac9d0d\") "
Oct 29 22:39:56.654123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:56.654087    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-openshift-ca" pod "d95156da-7686-41d2-b02f-54f07fac9d0d" (UID: "d95156da-7686-41d2-b02f-54f07fac9d0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:39:56.654706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:56.654638    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:56.655948 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:56.655916    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-xks-ca" pod "d95156da-7686-41d2-b02f-54f07fac9d0d" (UID: "d95156da-7686-41d2-b02f-54f07fac9d0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:39:56.656061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:56.656003    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95156da-7686-41d2-b02f-54f07fac9d0d-kube-api-access-r29z9" pod "d95156da-7686-41d2-b02f-54f07fac9d0d" (UID: "d95156da-7686-41d2-b02f-54f07fac9d0d"). InnerVolumeSpecName "kube-api-access-r29z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:39:56.756033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:56.756005    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r29z9\" (UniqueName: \"kubernetes.io/projected/d95156da-7686-41d2-b02f-54f07fac9d0d-kube-api-access-r29z9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:56.756033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:56.756033    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95156da-7686-41d2-b02f-54f07fac9d0d-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:39:57.051319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:57.051287    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:39:57.415950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:57.415870    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"
Oct 29 22:39:57.416125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:57.415870    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk" event={"ID":"d95156da-7686-41d2-b02f-54f07fac9d0d","Type":"ContainerDied","Data":"8c7cc4443dbaeed88c1eac080fa9737a7040897bafc641f4eb5195b4455f74d2"}
Oct 29 22:39:57.416125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:57.416005    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c7cc4443dbaeed88c1eac080fa9737a7040897bafc641f4eb5195b4455f74d2"
Oct 29 22:39:58.838013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:39:58.837984    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:00.229316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:00.229288    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:00.687404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:00.684606    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:00.836028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:00.835781    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:00.973010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:00.972801    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"]
Oct 29 22:40:00.973458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:00.973424    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx" podUID="b70e8c56-6765-4caa-8e1d-3eb08b7777d5" containerName="istio-proxy" containerID="cri-o://27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5" gracePeriod=30
Oct 29 22:40:01.069463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:01.069429    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:01.111347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:01.109742    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"]
Oct 29 22:40:01.117118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:01.116986    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a73b5848-w8bpl"]
Oct 29 22:40:01.199879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:01.199842    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:01.428250 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:01.428147    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"]
Oct 29 22:40:01.435142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:01.435111    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-616da7-nx5ns"]
Oct 29 22:40:01.807981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:01.807952    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:01.960531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:01.960486    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:02.092988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.092823    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:02.124881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.124843    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a600cd8-e011-46ab-9b3b-3f392d2afa19" path="/var/lib/kubelet/pods/2a600cd8-e011-46ab-9b3b-3f392d2afa19/volumes"
Oct 29 22:40:02.125729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.125705    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e63f6bb-b80e-424a-b3bb-83043d00f544" path="/var/lib/kubelet/pods/9e63f6bb-b80e-424a-b3bb-83043d00f544/volumes"
Oct 29 22:40:02.179648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.179606    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"]
Oct 29 22:40:02.192586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.192560    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee886f-49btn"]
Oct 29 22:40:02.215123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.215098    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:02.288501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.288464    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"]
Oct 29 22:40:02.290497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.290469    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d95156da-7686-41d2-b02f-54f07fac9d0d" containerName="revoke-keys"
Oct 29 22:40:02.290497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.290497    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95156da-7686-41d2-b02f-54f07fac9d0d" containerName="revoke-keys"
Oct 29 22:40:02.290989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.290966    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95156da-7686-41d2-b02f-54f07fac9d0d" containerName="revoke-keys"
Oct 29 22:40:02.312731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.312695    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:02.315227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.315200    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"]
Oct 29 22:40:02.329977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.329939    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:40:02.330181 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.330109    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:40:02.449889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.449857    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw\" (UID: \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:02.450365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.450040    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw\" (UID: \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:02.450365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.450234    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jhxz\" (UniqueName: \"kubernetes.io/projected/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-kube-api-access-5jhxz\") pod \"maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw\" (UID: \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:02.552116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.552073    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw\" (UID: \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:02.552370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.552351    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhxz\" (UniqueName: \"kubernetes.io/projected/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-kube-api-access-5jhxz\") pod \"maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw\" (UID: \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:02.552488 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.552471    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw\" (UID: \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:02.552880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.552854    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw\" (UID: \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:02.554949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.554925    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw\" (UID: \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:02.574423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.574400    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhxz\" (UniqueName: \"kubernetes.io/projected/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-kube-api-access-5jhxz\") pod \"maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw\" (UID: \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:02.630584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.630552    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:02.652517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.652490    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"]
Oct 29 22:40:02.680116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.678737    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:02.694412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.694382    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"]
Oct 29 22:40:02.822619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.822591    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:02.953775 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:40:02.953743    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d673cfd_c7ec_4f54_a96e_07904b39ab5d.slice/crio-40436123759aecbc35db1c5d7dad0e41d6d3f3857b2c47500a84e374bec51f2c WatchSource:0}: Error finding container 40436123759aecbc35db1c5d7dad0e41d6d3f3857b2c47500a84e374bec51f2c: Status 404 returned error can't find the container with id 40436123759aecbc35db1c5d7dad0e41d6d3f3857b2c47500a84e374bec51f2c
Oct 29 22:40:02.955632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.955601    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:02.956881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:02.956857    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"]
Oct 29 22:40:03.502515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:03.502484    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw" event={"ID":"9d673cfd-c7ec-4f54-a96e-07904b39ab5d","Type":"ContainerStarted","Data":"31991588a5583be1fd29de21600a5e5d63a633cdf98e35d6a8225128c072b4aa"}
Oct 29 22:40:03.502913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:03.502537    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw" event={"ID":"9d673cfd-c7ec-4f54-a96e-07904b39ab5d","Type":"ContainerStarted","Data":"40436123759aecbc35db1c5d7dad0e41d6d3f3857b2c47500a84e374bec51f2c"}
Oct 29 22:40:03.550257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:03.550170    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw" podStartSLOduration=1.550137782 podStartE2EDuration="1.550137782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:03.544391787 +0000 UTC m=+1545.569167938" watchObservedRunningTime="2026-10-29 22:40:03.550137782 +0000 UTC m=+1545.574913913"
Oct 29 22:40:04.116633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:04.116599    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e7ee26-39f3-4475-86ed-04a65e3e4f33" path="/var/lib/kubelet/pods/18e7ee26-39f3-4475-86ed-04a65e3e4f33/volumes"
Oct 29 22:40:04.182467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:04.182439    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:04.513728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:04.513694    4860 generic.go:356] "Generic (PLEG): container finished" podID="9d673cfd-c7ec-4f54-a96e-07904b39ab5d" containerID="31991588a5583be1fd29de21600a5e5d63a633cdf98e35d6a8225128c072b4aa" exitCode=0
Oct 29 22:40:04.514133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:04.513744    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw" event={"ID":"9d673cfd-c7ec-4f54-a96e-07904b39ab5d","Type":"ContainerDied","Data":"31991588a5583be1fd29de21600a5e5d63a633cdf98e35d6a8225128c072b4aa"}
Oct 29 22:40:05.399171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.399144    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"
Oct 29 22:40:05.399372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.399219    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"
Oct 29 22:40:05.708436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.708412    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:05.813649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.813613    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-xks-ca\") pod \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\" (UID: \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\") "
Oct 29 22:40:05.813817 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.813671    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-openshift-ca\") pod \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\" (UID: \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\") "
Oct 29 22:40:05.813817 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.813758    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-kube-api-access-5jhxz\" (UniqueName: \"kubernetes.io/projected/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-kube-api-access-5jhxz\") pod \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\" (UID: \"9d673cfd-c7ec-4f54-a96e-07904b39ab5d\") "
Oct 29 22:40:05.814161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.814125    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-openshift-ca" pod "9d673cfd-c7ec-4f54-a96e-07904b39ab5d" (UID: "9d673cfd-c7ec-4f54-a96e-07904b39ab5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:05.814531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.814506    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:05.816230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.816186    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-xks-ca" pod "9d673cfd-c7ec-4f54-a96e-07904b39ab5d" (UID: "9d673cfd-c7ec-4f54-a96e-07904b39ab5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:40:05.816599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.816576    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-kube-api-access-5jhxz" pod "9d673cfd-c7ec-4f54-a96e-07904b39ab5d" (UID: "9d673cfd-c7ec-4f54-a96e-07904b39ab5d"). InnerVolumeSpecName "kube-api-access-5jhxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:05.915918 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.915893    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:05.915918 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.915920    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jhxz\" (UniqueName: \"kubernetes.io/projected/9d673cfd-c7ec-4f54-a96e-07904b39ab5d-kube-api-access-5jhxz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:05.998906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:05.998878    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:06.262552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.262522    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:40:06.323566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.323537    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-data\") pod \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") "
Oct 29 22:40:06.323726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.323598    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-podinfo\") pod \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") "
Oct 29 22:40:06.323726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.323703    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-token\" (UniqueName: \"kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-token\") pod \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") "
Oct 29 22:40:06.323868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.323840    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-certs\") pod \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") "
Oct 29 22:40:06.323921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.323855    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-data" pod "b70e8c56-6765-4caa-8e1d-3eb08b7777d5" (UID: "b70e8c56-6765-4caa-8e1d-3eb08b7777d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:06.323921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.323872    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istiod-ca-cert\") pod \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") "
Oct 29 22:40:06.324019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.323986    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-envoy\") pod \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") "
Oct 29 22:40:06.324072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.324054    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-socket\") pod \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") "
Oct 29 22:40:06.324131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.324107    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-kube-api-access-d7jmw\" (UniqueName: \"kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-kube-api-access-d7jmw\") pod \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") "
Oct 29 22:40:06.324184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.324142    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-credential-socket\") pod \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\" (UID: \"b70e8c56-6765-4caa-8e1d-3eb08b7777d5\") "
Oct 29 22:40:06.324457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.324428    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istiod-ca-cert" pod "b70e8c56-6765-4caa-8e1d-3eb08b7777d5" (UID: "b70e8c56-6765-4caa-8e1d-3eb08b7777d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:06.324668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.324645    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-credential-socket" pod "b70e8c56-6765-4caa-8e1d-3eb08b7777d5" (UID: "b70e8c56-6765-4caa-8e1d-3eb08b7777d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:06.325107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.325078    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.325201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.325111    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.325201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.325126    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.325434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.325348    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-certs" pod "b70e8c56-6765-4caa-8e1d-3eb08b7777d5" (UID: "b70e8c56-6765-4caa-8e1d-3eb08b7777d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:06.325561 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.325539    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-socket" pod "b70e8c56-6765-4caa-8e1d-3eb08b7777d5" (UID: "b70e8c56-6765-4caa-8e1d-3eb08b7777d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:06.326736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.326704    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-token" pod "b70e8c56-6765-4caa-8e1d-3eb08b7777d5" (UID: "b70e8c56-6765-4caa-8e1d-3eb08b7777d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:06.327197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.327130    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-envoy" pod "b70e8c56-6765-4caa-8e1d-3eb08b7777d5" (UID: "b70e8c56-6765-4caa-8e1d-3eb08b7777d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:06.327831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.327804    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-kube-api-access-d7jmw" pod "b70e8c56-6765-4caa-8e1d-3eb08b7777d5" (UID: "b70e8c56-6765-4caa-8e1d-3eb08b7777d5"). InnerVolumeSpecName "kube-api-access-d7jmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:06.328165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.328148    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-podinfo" pod "b70e8c56-6765-4caa-8e1d-3eb08b7777d5" (UID: "b70e8c56-6765-4caa-8e1d-3eb08b7777d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:06.426157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.426119    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.426157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.426152    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.426414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.426165    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7jmw\" (UniqueName: \"kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-kube-api-access-d7jmw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.426414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.426179    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.426414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.426190    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.426414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.426200    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b70e8c56-6765-4caa-8e1d-3eb08b7777d5-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:06.538007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.537932    4860 generic.go:356] "Generic (PLEG): container finished" podID="b70e8c56-6765-4caa-8e1d-3eb08b7777d5" containerID="27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5" exitCode=0
Oct 29 22:40:06.538163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.538034    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"
Oct 29 22:40:06.538163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.538065    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx" event={"ID":"b70e8c56-6765-4caa-8e1d-3eb08b7777d5","Type":"ContainerDied","Data":"27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5"}
Oct 29 22:40:06.538163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.538119    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx" event={"ID":"b70e8c56-6765-4caa-8e1d-3eb08b7777d5","Type":"ContainerDied","Data":"46ff67258a04c8481bbcaa46d17df89a39fe5376c4ed0fc52066de953c537afd"}
Oct 29 22:40:06.538163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.538141    4860 scope.go:122] "RemoveContainer" containerID="27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5"
Oct 29 22:40:06.540125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.540108    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"
Oct 29 22:40:06.540226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.540139    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw" event={"ID":"9d673cfd-c7ec-4f54-a96e-07904b39ab5d","Type":"ContainerDied","Data":"40436123759aecbc35db1c5d7dad0e41d6d3f3857b2c47500a84e374bec51f2c"}
Oct 29 22:40:06.540226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.540187    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40436123759aecbc35db1c5d7dad0e41d6d3f3857b2c47500a84e374bec51f2c"
Oct 29 22:40:06.551115 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.551099    4860 scope.go:122] "RemoveContainer" containerID="27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5"
Oct 29 22:40:06.551374 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:40:06.551352    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5\": container with ID starting with 27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5 not found: ID does not exist" containerID="27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5"
Oct 29 22:40:06.551443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.551384    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5"} err="failed to get container status \"27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5\": rpc error: code = NotFound desc = could not find container \"27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5\": container with ID starting with 27e7997b4395e6b21836594d3ab36cf746a330d2d35ac55bfe27f107d69804d5 not found: ID does not exist"
Oct 29 22:40:06.615630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.615596    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"]
Oct 29 22:40:06.620676 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:06.620653    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cae55b-openshift-default-f795f5d7f-6x2tx"]
Oct 29 22:40:07.655149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:07.652974    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg" podUID="46dfd5e5-280c-4064-95cf-1b3162c50cb2" containerName="payload-processing" containerID="cri-o://c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe" gracePeriod=25
Oct 29 22:40:07.693732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:07.693702    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:07.695506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:07.695482    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf" podUID="099d59a7-42df-4429-a904-61fb44a67d79" containerName="payload-pre-processing" containerID="cri-o://80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6" gracePeriod=25
Oct 29 22:40:08.004756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.004726    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"
Oct 29 22:40:08.049780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.048593    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46dfd5e5-280c-4064-95cf-1b3162c50cb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46dfd5e5-280c-4064-95cf-1b3162c50cb2-plugins-config-volume\") pod \"46dfd5e5-280c-4064-95cf-1b3162c50cb2\" (UID: \"46dfd5e5-280c-4064-95cf-1b3162c50cb2\") "
Oct 29 22:40:08.049780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.048734    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46dfd5e5-280c-4064-95cf-1b3162c50cb2-kube-api-access-pk2sw\" (UniqueName: \"kubernetes.io/projected/46dfd5e5-280c-4064-95cf-1b3162c50cb2-kube-api-access-pk2sw\") pod \"46dfd5e5-280c-4064-95cf-1b3162c50cb2\" (UID: \"46dfd5e5-280c-4064-95cf-1b3162c50cb2\") "
Oct 29 22:40:08.052305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.052270    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46dfd5e5-280c-4064-95cf-1b3162c50cb2-kube-api-access-pk2sw" pod "46dfd5e5-280c-4064-95cf-1b3162c50cb2" (UID: "46dfd5e5-280c-4064-95cf-1b3162c50cb2"). InnerVolumeSpecName "kube-api-access-pk2sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:08.052683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.052659    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dfd5e5-280c-4064-95cf-1b3162c50cb2-plugins-config-volume" pod "46dfd5e5-280c-4064-95cf-1b3162c50cb2" (UID: "46dfd5e5-280c-4064-95cf-1b3162c50cb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:08.071851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.071823    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"
Oct 29 22:40:08.114019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.113991    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70e8c56-6765-4caa-8e1d-3eb08b7777d5" path="/var/lib/kubelet/pods/b70e8c56-6765-4caa-8e1d-3eb08b7777d5/volumes"
Oct 29 22:40:08.150748 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.150716    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099d59a7-42df-4429-a904-61fb44a67d79-kube-api-access-tfl9n\" (UniqueName: \"kubernetes.io/projected/099d59a7-42df-4429-a904-61fb44a67d79-kube-api-access-tfl9n\") pod \"099d59a7-42df-4429-a904-61fb44a67d79\" (UID: \"099d59a7-42df-4429-a904-61fb44a67d79\") "
Oct 29 22:40:08.151002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.150986    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/099d59a7-42df-4429-a904-61fb44a67d79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/099d59a7-42df-4429-a904-61fb44a67d79-plugins-config-volume\") pod \"099d59a7-42df-4429-a904-61fb44a67d79\" (UID: \"099d59a7-42df-4429-a904-61fb44a67d79\") "
Oct 29 22:40:08.151779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.151751    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46dfd5e5-280c-4064-95cf-1b3162c50cb2-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:08.151779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.151779    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk2sw\" (UniqueName: \"kubernetes.io/projected/46dfd5e5-280c-4064-95cf-1b3162c50cb2-kube-api-access-pk2sw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:08.155190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.155157    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099d59a7-42df-4429-a904-61fb44a67d79-plugins-config-volume" pod "099d59a7-42df-4429-a904-61fb44a67d79" (UID: "099d59a7-42df-4429-a904-61fb44a67d79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:08.155672 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.155649    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099d59a7-42df-4429-a904-61fb44a67d79-kube-api-access-tfl9n" pod "099d59a7-42df-4429-a904-61fb44a67d79" (UID: "099d59a7-42df-4429-a904-61fb44a67d79"). InnerVolumeSpecName "kube-api-access-tfl9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:08.252737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.252707    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/099d59a7-42df-4429-a904-61fb44a67d79-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:08.252737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.252738    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfl9n\" (UniqueName: \"kubernetes.io/projected/099d59a7-42df-4429-a904-61fb44a67d79-kube-api-access-tfl9n\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:08.524290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.524215    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:08.589581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.585876    4860 generic.go:356] "Generic (PLEG): container finished" podID="099d59a7-42df-4429-a904-61fb44a67d79" containerID="80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6" exitCode=0
Oct 29 22:40:08.589581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.585998    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf" event={"ID":"099d59a7-42df-4429-a904-61fb44a67d79","Type":"ContainerDied","Data":"80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6"}
Oct 29 22:40:08.589581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.586089    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf" event={"ID":"099d59a7-42df-4429-a904-61fb44a67d79","Type":"ContainerDied","Data":"21352f407d6de50eb7d14088f6efc5c1ebd156bab5696c28dead75871061a4f6"}
Oct 29 22:40:08.589581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.586121    4860 scope.go:122] "RemoveContainer" containerID="80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6"
Oct 29 22:40:08.589581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.586445    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"
Oct 29 22:40:08.591265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.591227    4860 generic.go:356] "Generic (PLEG): container finished" podID="46dfd5e5-280c-4064-95cf-1b3162c50cb2" containerID="c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe" exitCode=0
Oct 29 22:40:08.591469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.591446    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg" event={"ID":"46dfd5e5-280c-4064-95cf-1b3162c50cb2","Type":"ContainerDied","Data":"c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe"}
Oct 29 22:40:08.591513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.591505    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg" event={"ID":"46dfd5e5-280c-4064-95cf-1b3162c50cb2","Type":"ContainerDied","Data":"5465394a62d59b6d5c0d3fc3c063338e6d11a84596c9e9dd5736ed9619c32f6a"}
Oct 29 22:40:08.591609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.591597    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"
Oct 29 22:40:08.608104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.608077    4860 scope.go:122] "RemoveContainer" containerID="80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6"
Oct 29 22:40:08.608477 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:40:08.608450    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6\": container with ID starting with 80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6 not found: ID does not exist" containerID="80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6"
Oct 29 22:40:08.608568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.608484    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6"} err="failed to get container status \"80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6\": rpc error: code = NotFound desc = could not find container \"80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6\": container with ID starting with 80e560deab91b6169c7fb2e9085d0d0b0fbd569c4bd4e05340ae9cc8bd0471a6 not found: ID does not exist"
Oct 29 22:40:08.608568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.608501    4860 scope.go:122] "RemoveContainer" containerID="c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe"
Oct 29 22:40:08.623469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.623444    4860 scope.go:122] "RemoveContainer" containerID="c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe"
Oct 29 22:40:08.624007 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:40:08.623971    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe\": container with ID starting with c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe not found: ID does not exist" containerID="c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe"
Oct 29 22:40:08.624157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.624014    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe"} err="failed to get container status \"c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe\": rpc error: code = NotFound desc = could not find container \"c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe\": container with ID starting with c3f777e0c0b913cf27a1549605d4b334cdd836c45d743604a6180c4e7c3faebe not found: ID does not exist"
Oct 29 22:40:08.651909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.651872    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"]
Oct 29 22:40:08.661293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.661264    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:08.663588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.663536    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa4b8b6-675f9bbd4c-4jhnf"]
Oct 29 22:40:08.691376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.691351    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"]
Oct 29 22:40:08.696708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.696683    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa4b8b6-68f88d6f68-2sqfg"]
Oct 29 22:40:08.813382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.813276    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:08.971876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:08.971840    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:09.425982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.425936    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:09.441078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.441047    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"]
Oct 29 22:40:09.441855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.441839    4860 cpu_manager.go:414] "containerMap: removing container" podUID="46dfd5e5-280c-4064-95cf-1b3162c50cb2" containerName="payload-processing"
Oct 29 22:40:09.441901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.441856    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46dfd5e5-280c-4064-95cf-1b3162c50cb2" containerName="payload-processing"
Oct 29 22:40:09.441901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.441874    4860 cpu_manager.go:414] "containerMap: removing container" podUID="099d59a7-42df-4429-a904-61fb44a67d79" containerName="payload-pre-processing"
Oct 29 22:40:09.441901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.441879    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099d59a7-42df-4429-a904-61fb44a67d79" containerName="payload-pre-processing"
Oct 29 22:40:09.441901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.441895    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b70e8c56-6765-4caa-8e1d-3eb08b7777d5" containerName="istio-proxy"
Oct 29 22:40:09.441901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.441899    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70e8c56-6765-4caa-8e1d-3eb08b7777d5" containerName="istio-proxy"
Oct 29 22:40:09.442077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.441921    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9d673cfd-c7ec-4f54-a96e-07904b39ab5d" containerName="revoke-keys"
Oct 29 22:40:09.442077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.441926    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d673cfd-c7ec-4f54-a96e-07904b39ab5d" containerName="revoke-keys"
Oct 29 22:40:09.442077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.442042    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d673cfd-c7ec-4f54-a96e-07904b39ab5d" containerName="revoke-keys"
Oct 29 22:40:09.442077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.442059    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="46dfd5e5-280c-4064-95cf-1b3162c50cb2" containerName="payload-processing"
Oct 29 22:40:09.442077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.442065    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="099d59a7-42df-4429-a904-61fb44a67d79" containerName="payload-pre-processing"
Oct 29 22:40:09.442077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.442075    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70e8c56-6765-4caa-8e1d-3eb08b7777d5" containerName="istio-proxy"
Oct 29 22:40:09.445968 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.445951    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:09.470972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.467258    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"]
Oct 29 22:40:09.476466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.473582    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:40:09.476466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.473633    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:40:09.493420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.493385    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-39322f-9z97s\" (UID: \"52dec074-2dee-4aa2-962e-66be90f779b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:09.493678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.493442    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-39322f-9z97s\" (UID: \"52dec074-2dee-4aa2-962e-66be90f779b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:09.493770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.493678    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkg96\" (UniqueName: \"kubernetes.io/projected/52dec074-2dee-4aa2-962e-66be90f779b2-kube-api-access-qkg96\") pod \"maas-api-revoke-keys-e2e-api-b-39322f-9z97s\" (UID: \"52dec074-2dee-4aa2-962e-66be90f779b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:09.554731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.554696    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:09.595622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.595595    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-39322f-9z97s\" (UID: \"52dec074-2dee-4aa2-962e-66be90f779b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:09.595801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.595647    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-39322f-9z97s\" (UID: \"52dec074-2dee-4aa2-962e-66be90f779b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:09.595801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.595704    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkg96\" (UniqueName: \"kubernetes.io/projected/52dec074-2dee-4aa2-962e-66be90f779b2-kube-api-access-qkg96\") pod \"maas-api-revoke-keys-e2e-api-b-39322f-9z97s\" (UID: \"52dec074-2dee-4aa2-962e-66be90f779b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:09.596417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.596389    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-39322f-9z97s\" (UID: \"52dec074-2dee-4aa2-962e-66be90f779b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:09.598295 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.598270    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-39322f-9z97s\" (UID: \"52dec074-2dee-4aa2-962e-66be90f779b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:09.616475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.616451    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkg96\" (UniqueName: \"kubernetes.io/projected/52dec074-2dee-4aa2-962e-66be90f779b2-kube-api-access-qkg96\") pod \"maas-api-revoke-keys-e2e-api-b-39322f-9z97s\" (UID: \"52dec074-2dee-4aa2-962e-66be90f779b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:09.683729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.683696    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:09.714066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.714034    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"]
Oct 29 22:40:09.714279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.714253    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy" containerID="cri-o://8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa" gracePeriod=30
Oct 29 22:40:09.765374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.765348    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:09.807887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.807859    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:09.831468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.828017    4860 patch_prober.go:29] interesting pod/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:40:09.831468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:09.828077    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:40:10.116179 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.116147    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="099d59a7-42df-4429-a904-61fb44a67d79" path="/var/lib/kubelet/pods/099d59a7-42df-4429-a904-61fb44a67d79/volumes"
Oct 29 22:40:10.116629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.116613    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46dfd5e5-280c-4064-95cf-1b3162c50cb2" path="/var/lib/kubelet/pods/46dfd5e5-280c-4064-95cf-1b3162c50cb2/volumes"
Oct 29 22:40:10.178180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.178150    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"]
Oct 29 22:40:10.261737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.261661    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"]
Oct 29 22:40:10.268220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.268198    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:10.285643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.283930    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"]
Oct 29 22:40:10.310402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.310370    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d1777-ng489\" (UID: \"e56007fe-52ad-46e2-845d-681a239a8bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:10.310594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.310554    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxjqc\" (UniqueName: \"kubernetes.io/projected/e56007fe-52ad-46e2-845d-681a239a8bf3-kube-api-access-mxjqc\") pod \"maas-api-revoke-keys-e2e-api-b-3d1777-ng489\" (UID: \"e56007fe-52ad-46e2-845d-681a239a8bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:10.310737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.310711    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d1777-ng489\" (UID: \"e56007fe-52ad-46e2-845d-681a239a8bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:10.412636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.412599    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d1777-ng489\" (UID: \"e56007fe-52ad-46e2-845d-681a239a8bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:10.412923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.412708    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxjqc\" (UniqueName: \"kubernetes.io/projected/e56007fe-52ad-46e2-845d-681a239a8bf3-kube-api-access-mxjqc\") pod \"maas-api-revoke-keys-e2e-api-b-3d1777-ng489\" (UID: \"e56007fe-52ad-46e2-845d-681a239a8bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:10.412923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.412821    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d1777-ng489\" (UID: \"e56007fe-52ad-46e2-845d-681a239a8bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:10.413253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.413231    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d1777-ng489\" (UID: \"e56007fe-52ad-46e2-845d-681a239a8bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:10.415186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.415166    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3d1777-ng489\" (UID: \"e56007fe-52ad-46e2-845d-681a239a8bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:10.434250 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.434223    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxjqc\" (UniqueName: \"kubernetes.io/projected/e56007fe-52ad-46e2-845d-681a239a8bf3-kube-api-access-mxjqc\") pod \"maas-api-revoke-keys-e2e-api-b-3d1777-ng489\" (UID: \"e56007fe-52ad-46e2-845d-681a239a8bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:10.599991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.599971    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:10.617338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.617296    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s" event={"ID":"52dec074-2dee-4aa2-962e-66be90f779b2","Type":"ContainerStarted","Data":"2f180b9c7ae1a36984789c166bc995b5b894bd345e8ea0314210080fea7694d7"}
Oct 29 22:40:10.617462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.617390    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s" event={"ID":"52dec074-2dee-4aa2-962e-66be90f779b2","Type":"ContainerStarted","Data":"97dab08a9c135632e418c099369177c7cb011bf4daa599ba2f099ed226be165c"}
Oct 29 22:40:10.661671 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.661614    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s" podStartSLOduration=1.661595273 podStartE2EDuration="1.661595273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:10.660451936 +0000 UTC m=+1552.685228069" watchObservedRunningTime="2026-10-29 22:40:10.661595273 +0000 UTC m=+1552.686371407"
Oct 29 22:40:10.908307 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:40:10.908279    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56007fe_52ad_46e2_845d_681a239a8bf3.slice/crio-aaf6edf45d905ae2e83394930aae9c78bf6abd97705a38faf1cc76e7de6bd28b WatchSource:0}: Error finding container aaf6edf45d905ae2e83394930aae9c78bf6abd97705a38faf1cc76e7de6bd28b: Status 404 returned error can't find the container with id aaf6edf45d905ae2e83394930aae9c78bf6abd97705a38faf1cc76e7de6bd28b
Oct 29 22:40:10.912338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:10.912297    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"]
Oct 29 22:40:11.033919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:11.033881    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"]
Oct 29 22:40:11.045489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:11.041948    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db144c1c-tzcw5"]
Oct 29 22:40:11.629058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:11.629022    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489" event={"ID":"e56007fe-52ad-46e2-845d-681a239a8bf3","Type":"ContainerStarted","Data":"f417ad30dc4d01dbd5a11ff17e855ecb7aacd767ca84a56f92b7081ed4b5f2cc"}
Oct 29 22:40:11.629260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:11.629088    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489" event={"ID":"e56007fe-52ad-46e2-845d-681a239a8bf3","Type":"ContainerStarted","Data":"aaf6edf45d905ae2e83394930aae9c78bf6abd97705a38faf1cc76e7de6bd28b"}
Oct 29 22:40:11.631073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:11.631050    4860 generic.go:356] "Generic (PLEG): container finished" podID="52dec074-2dee-4aa2-962e-66be90f779b2" containerID="2f180b9c7ae1a36984789c166bc995b5b894bd345e8ea0314210080fea7694d7" exitCode=0
Oct 29 22:40:11.631182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:11.631139    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s" event={"ID":"52dec074-2dee-4aa2-962e-66be90f779b2","Type":"ContainerDied","Data":"2f180b9c7ae1a36984789c166bc995b5b894bd345e8ea0314210080fea7694d7"}
Oct 29 22:40:11.637713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:11.637695    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:11.687273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:11.684259    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489" podStartSLOduration=1.684236877 podStartE2EDuration="1.684236877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:11.679839515 +0000 UTC m=+1553.704615669" watchObservedRunningTime="2026-10-29 22:40:11.684236877 +0000 UTC m=+1553.709013009"
Oct 29 22:40:12.113359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.113310    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32c0fdd2-4ee7-4b7b-8432-cda376900576" path="/var/lib/kubelet/pods/32c0fdd2-4ee7-4b7b-8432-cda376900576/volumes"
Oct 29 22:40:12.180550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.180516    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:12.298514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.298434    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:12.643008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.642908    4860 generic.go:356] "Generic (PLEG): container finished" podID="e56007fe-52ad-46e2-845d-681a239a8bf3" containerID="f417ad30dc4d01dbd5a11ff17e855ecb7aacd767ca84a56f92b7081ed4b5f2cc" exitCode=0
Oct 29 22:40:12.643008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.642985    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489" event={"ID":"e56007fe-52ad-46e2-845d-681a239a8bf3","Type":"ContainerDied","Data":"f417ad30dc4d01dbd5a11ff17e855ecb7aacd767ca84a56f92b7081ed4b5f2cc"}
Oct 29 22:40:12.769858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.769801    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"]
Oct 29 22:40:12.835408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.835377    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:12.899937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.899856    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:12.953486 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.953451    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52dec074-2dee-4aa2-962e-66be90f779b2-kube-api-access-qkg96\" (UniqueName: \"kubernetes.io/projected/52dec074-2dee-4aa2-962e-66be90f779b2-kube-api-access-qkg96\") pod \"52dec074-2dee-4aa2-962e-66be90f779b2\" (UID: \"52dec074-2dee-4aa2-962e-66be90f779b2\") "
Oct 29 22:40:12.953702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.953684    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-xks-ca\") pod \"52dec074-2dee-4aa2-962e-66be90f779b2\" (UID: \"52dec074-2dee-4aa2-962e-66be90f779b2\") "
Oct 29 22:40:12.953901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.953884    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-openshift-ca\") pod \"52dec074-2dee-4aa2-962e-66be90f779b2\" (UID: \"52dec074-2dee-4aa2-962e-66be90f779b2\") "
Oct 29 22:40:12.954286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.954233    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-openshift-ca" pod "52dec074-2dee-4aa2-962e-66be90f779b2" (UID: "52dec074-2dee-4aa2-962e-66be90f779b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:12.954667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.954638    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:12.955672 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.955650    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52dec074-2dee-4aa2-962e-66be90f779b2-kube-api-access-qkg96" pod "52dec074-2dee-4aa2-962e-66be90f779b2" (UID: "52dec074-2dee-4aa2-962e-66be90f779b2"). InnerVolumeSpecName "kube-api-access-qkg96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:12.955887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:12.955863    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-xks-ca" pod "52dec074-2dee-4aa2-962e-66be90f779b2" (UID: "52dec074-2dee-4aa2-962e-66be90f779b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:40:13.030792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.030759    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:13.056188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.056151    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dec074-2dee-4aa2-962e-66be90f779b2-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:13.056188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.056187    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkg96\" (UniqueName: \"kubernetes.io/projected/52dec074-2dee-4aa2-962e-66be90f779b2-kube-api-access-qkg96\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:13.132302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.132271    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"]
Oct 29 22:40:13.133141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.133124    4860 cpu_manager.go:414] "containerMap: removing container" podUID="52dec074-2dee-4aa2-962e-66be90f779b2" containerName="revoke-keys"
Oct 29 22:40:13.133189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.133143    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52dec074-2dee-4aa2-962e-66be90f779b2" containerName="revoke-keys"
Oct 29 22:40:13.133289 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.133280    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="52dec074-2dee-4aa2-962e-66be90f779b2" containerName="revoke-keys"
Oct 29 22:40:13.137114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.137096    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.155338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.155263    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c296a3-serving-cert\""
Oct 29 22:40:13.162847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.162824    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"]
Oct 29 22:40:13.247954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.247919    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:13.258937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.258904    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fff808-0950-42f8-98dd-efb9c6a8fbac-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.259054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.258946    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-maas-api-tls\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.259054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.259007    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-metrics-certs\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.259180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.259145    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvvwx\" (UniqueName: \"kubernetes.io/projected/69fff808-0950-42f8-98dd-efb9c6a8fbac-kube-api-access-vvvwx\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.360922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.360898    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fff808-0950-42f8-98dd-efb9c6a8fbac-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.361059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.360930    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-maas-api-tls\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.361059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.360951    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-metrics-certs\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.361059 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:40:13.361046    4860 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-serving-cert: secret "maas-api-e2e-inf-b-c296a3-serving-cert" not found
Oct 29 22:40:13.361181 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.361078    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvvwx\" (UniqueName: \"kubernetes.io/projected/69fff808-0950-42f8-98dd-efb9c6a8fbac-kube-api-access-vvvwx\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.361181 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:40:13.361102    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-maas-api-tls podName:69fff808-0950-42f8-98dd-efb9c6a8fbac nodeName:}" failed. No retries permitted until 2026-10-29 22:40:13.861087568 +0000 UTC m=+1555.885863676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-maas-api-tls") pod "maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9" (UID: "69fff808-0950-42f8-98dd-efb9c6a8fbac") : secret "maas-api-e2e-inf-b-c296a3-serving-cert" not found
Oct 29 22:40:13.361682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.361662    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fff808-0950-42f8-98dd-efb9c6a8fbac-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.363474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.363454    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-metrics-certs\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.376368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.376319    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:13.382301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.382279    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvvwx\" (UniqueName: \"kubernetes.io/projected/69fff808-0950-42f8-98dd-efb9c6a8fbac-kube-api-access-vvvwx\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.501182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.501078    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:13.656263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.656225    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"
Oct 29 22:40:13.656440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.656256    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s" event={"ID":"52dec074-2dee-4aa2-962e-66be90f779b2","Type":"ContainerDied","Data":"97dab08a9c135632e418c099369177c7cb011bf4daa599ba2f099ed226be165c"}
Oct 29 22:40:13.656440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.656316    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97dab08a9c135632e418c099369177c7cb011bf4daa599ba2f099ed226be165c"
Oct 29 22:40:13.885044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.884996    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-maas-api-tls\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.888751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.888718    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-maas-api-tls\") pod \"maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:13.897495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.897464    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:13.986241 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.986206    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-openshift-ca\") pod \"e56007fe-52ad-46e2-845d-681a239a8bf3\" (UID: \"e56007fe-52ad-46e2-845d-681a239a8bf3\") "
Oct 29 22:40:13.986576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.986549    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56007fe-52ad-46e2-845d-681a239a8bf3-kube-api-access-mxjqc\" (UniqueName: \"kubernetes.io/projected/e56007fe-52ad-46e2-845d-681a239a8bf3-kube-api-access-mxjqc\") pod \"e56007fe-52ad-46e2-845d-681a239a8bf3\" (UID: \"e56007fe-52ad-46e2-845d-681a239a8bf3\") "
Oct 29 22:40:13.986784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.986615    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-xks-ca\") pod \"e56007fe-52ad-46e2-845d-681a239a8bf3\" (UID: \"e56007fe-52ad-46e2-845d-681a239a8bf3\") "
Oct 29 22:40:13.987309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.987244    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-openshift-ca" pod "e56007fe-52ad-46e2-845d-681a239a8bf3" (UID: "e56007fe-52ad-46e2-845d-681a239a8bf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:13.988989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.988964    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-xks-ca" pod "e56007fe-52ad-46e2-845d-681a239a8bf3" (UID: "e56007fe-52ad-46e2-845d-681a239a8bf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:40:13.989647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:13.989627    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56007fe-52ad-46e2-845d-681a239a8bf3-kube-api-access-mxjqc" pod "e56007fe-52ad-46e2-845d-681a239a8bf3" (UID: "e56007fe-52ad-46e2-845d-681a239a8bf3"). InnerVolumeSpecName "kube-api-access-mxjqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:14.054799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:14.054749    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:14.089180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:14.089114    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:14.089180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:14.089151    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxjqc\" (UniqueName: \"kubernetes.io/projected/e56007fe-52ad-46e2-845d-681a239a8bf3-kube-api-access-mxjqc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:14.089180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:14.089167    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56007fe-52ad-46e2-845d-681a239a8bf3-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:14.383553 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:40:14.383514    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fff808_0950_42f8_98dd_efb9c6a8fbac.slice/crio-b3529908187f2616f172626076bc878f8cad14b10645742768795038d960fdaa WatchSource:0}: Error finding container b3529908187f2616f172626076bc878f8cad14b10645742768795038d960fdaa: Status 404 returned error can't find the container with id b3529908187f2616f172626076bc878f8cad14b10645742768795038d960fdaa
Oct 29 22:40:14.450705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:14.450656    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"]
Oct 29 22:40:14.667735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:14.667650    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9" event={"ID":"69fff808-0950-42f8-98dd-efb9c6a8fbac","Type":"ContainerStarted","Data":"b3529908187f2616f172626076bc878f8cad14b10645742768795038d960fdaa"}
Oct 29 22:40:14.669239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:14.669218    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"
Oct 29 22:40:14.669239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:14.669221    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489" event={"ID":"e56007fe-52ad-46e2-845d-681a239a8bf3","Type":"ContainerDied","Data":"aaf6edf45d905ae2e83394930aae9c78bf6abd97705a38faf1cc76e7de6bd28b"}
Oct 29 22:40:14.669475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:14.669269    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaf6edf45d905ae2e83394930aae9c78bf6abd97705a38faf1cc76e7de6bd28b"
Oct 29 22:40:15.045285 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.045256    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:15.046608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.046583    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:40:15.236630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.236598    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-data\") pod \"551bb653-c966-43cc-8f02-007606849b1f\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") "
Oct 29 22:40:15.236796 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.236668    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/551bb653-c966-43cc-8f02-007606849b1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551bb653-c966-43cc-8f02-007606849b1f-istio-podinfo\") pod \"551bb653-c966-43cc-8f02-007606849b1f\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") "
Oct 29 22:40:15.236841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.236791    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-socket\") pod \"551bb653-c966-43cc-8f02-007606849b1f\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") "
Oct 29 22:40:15.236881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.236853    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-kube-api-access-4db85\" (UniqueName: \"kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-kube-api-access-4db85\") pod \"551bb653-c966-43cc-8f02-007606849b1f\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") "
Oct 29 22:40:15.236936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.236874    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-data" pod "551bb653-c966-43cc-8f02-007606849b1f" (UID: "551bb653-c966-43cc-8f02-007606849b1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:15.236936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.236920    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-credential-socket\") pod \"551bb653-c966-43cc-8f02-007606849b1f\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") "
Oct 29 22:40:15.237065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.236972    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-istio-token\" (UniqueName: \"kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-istio-token\") pod \"551bb653-c966-43cc-8f02-007606849b1f\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") "
Oct 29 22:40:15.237127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.237059    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-socket" pod "551bb653-c966-43cc-8f02-007606849b1f" (UID: "551bb653-c966-43cc-8f02-007606849b1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:15.237187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.237167    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551bb653-c966-43cc-8f02-007606849b1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551bb653-c966-43cc-8f02-007606849b1f-istiod-ca-cert\") pod \"551bb653-c966-43cc-8f02-007606849b1f\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") "
Oct 29 22:40:15.237278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.237259    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-certs\") pod \"551bb653-c966-43cc-8f02-007606849b1f\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") "
Oct 29 22:40:15.237411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.237296    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-envoy\") pod \"551bb653-c966-43cc-8f02-007606849b1f\" (UID: \"551bb653-c966-43cc-8f02-007606849b1f\") "
Oct 29 22:40:15.237411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.237304    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-credential-socket" pod "551bb653-c966-43cc-8f02-007606849b1f" (UID: "551bb653-c966-43cc-8f02-007606849b1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:15.237609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.237583    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551bb653-c966-43cc-8f02-007606849b1f-istiod-ca-cert" pod "551bb653-c966-43cc-8f02-007606849b1f" (UID: "551bb653-c966-43cc-8f02-007606849b1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:15.237674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.237588    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-certs" pod "551bb653-c966-43cc-8f02-007606849b1f" (UID: "551bb653-c966-43cc-8f02-007606849b1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:15.238085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.238065    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:15.238085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.238086    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:15.238236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.238096    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551bb653-c966-43cc-8f02-007606849b1f-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:15.238236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.238104    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:15.238236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.238113    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:15.239307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.239279    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-kube-api-access-4db85" pod "551bb653-c966-43cc-8f02-007606849b1f" (UID: "551bb653-c966-43cc-8f02-007606849b1f"). InnerVolumeSpecName "kube-api-access-4db85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:15.239624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.239597    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/551bb653-c966-43cc-8f02-007606849b1f-istio-podinfo" pod "551bb653-c966-43cc-8f02-007606849b1f" (UID: "551bb653-c966-43cc-8f02-007606849b1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:15.239822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.239803    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-envoy" pod "551bb653-c966-43cc-8f02-007606849b1f" (UID: "551bb653-c966-43cc-8f02-007606849b1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:15.239904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.239816    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-istio-token" pod "551bb653-c966-43cc-8f02-007606849b1f" (UID: "551bb653-c966-43cc-8f02-007606849b1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:15.340123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.340055    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4db85\" (UniqueName: \"kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-kube-api-access-4db85\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:15.340123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.340086    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551bb653-c966-43cc-8f02-007606849b1f-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:15.340123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.340096    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551bb653-c966-43cc-8f02-007606849b1f-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:15.340123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.340107    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551bb653-c966-43cc-8f02-007606849b1f-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:15.679578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.679492    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9" event={"ID":"69fff808-0950-42f8-98dd-efb9c6a8fbac","Type":"ContainerStarted","Data":"d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264"}
Oct 29 22:40:15.679578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.679567    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:15.681438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.681406    4860 generic.go:356] "Generic (PLEG): container finished" podID="551bb653-c966-43cc-8f02-007606849b1f" containerID="8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa" exitCode=0
Oct 29 22:40:15.681579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.681443    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"
Oct 29 22:40:15.681579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.681487    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" event={"ID":"551bb653-c966-43cc-8f02-007606849b1f","Type":"ContainerDied","Data":"8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa"}
Oct 29 22:40:15.681579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.681528    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s" event={"ID":"551bb653-c966-43cc-8f02-007606849b1f","Type":"ContainerDied","Data":"d00312cd0e4a4aa7ba135dcfc217eaedd874903a68840195b0187fe1accf0712"}
Oct 29 22:40:15.681579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.681549    4860 scope.go:122] "RemoveContainer" containerID="8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa"
Oct 29 22:40:15.700414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.700395    4860 scope.go:122] "RemoveContainer" containerID="0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813"
Oct 29 22:40:15.712122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.712104    4860 scope.go:122] "RemoveContainer" containerID="8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa"
Oct 29 22:40:15.712413 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:40:15.712395    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa\": container with ID starting with 8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa not found: ID does not exist" containerID="8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa"
Oct 29 22:40:15.712474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.712423    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa"} err="failed to get container status \"8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa\": rpc error: code = NotFound desc = could not find container \"8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa\": container with ID starting with 8fe0a59beac6443a3456c82f9a555f147c431fac5537127882540e59c47afdfa not found: ID does not exist"
Oct 29 22:40:15.712474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.712441    4860 scope.go:122] "RemoveContainer" containerID="0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813"
Oct 29 22:40:15.712678 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:40:15.712663    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813\": container with ID starting with 0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813 not found: ID does not exist" containerID="0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813"
Oct 29 22:40:15.712720 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.712681    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813"} err="failed to get container status \"0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813\": rpc error: code = NotFound desc = could not find container \"0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813\": container with ID starting with 0e3a5de26ec99f231557ef8e14194ccb47d9b8bce9b6498b1a49ec9643260813 not found: ID does not exist"
Oct 29 22:40:15.726927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.726874    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9" podStartSLOduration=2.336390218 podStartE2EDuration="2.726860781s" totalImagesPullingTime="390.470563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:15.725215031 +0000 UTC m=+1557.749991159" watchObservedRunningTime="2026-10-29 22:40:15.726860781 +0000 UTC m=+1557.751636978"
Oct 29 22:40:15.768802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.768779    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"]
Oct 29 22:40:15.773735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.773711    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ee886f-openshift-default-7f66f8dfb5-m9m5s"]
Oct 29 22:40:15.786534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:15.786513    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:16.114463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:16.114430    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551bb653-c966-43cc-8f02-007606849b1f" path="/var/lib/kubelet/pods/551bb653-c966-43cc-8f02-007606849b1f/volumes"
Oct 29 22:40:16.460084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:16.460055    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:17.154766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:17.154725    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:17.270594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:17.270553    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"]
Oct 29 22:40:17.285013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:17.284980    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa4b8b6-ps4wk"]
Oct 29 22:40:17.302425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:17.299694    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:17.770913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:17.770866    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n" podUID="26b8069e-48b2-463d-9268-7ad8298922a7" containerName="payload-pre-processing" containerID="cri-o://fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4" gracePeriod=25
Oct 29 22:40:17.792488 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:17.792449    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:18.060179 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:40:18.060139    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:40:18.081644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.081622    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"
Oct 29 22:40:18.115954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.115929    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95156da-7686-41d2-b02f-54f07fac9d0d" path="/var/lib/kubelet/pods/d95156da-7686-41d2-b02f-54f07fac9d0d/volumes"
Oct 29 22:40:18.200813 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.200784    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b8069e-48b2-463d-9268-7ad8298922a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b8069e-48b2-463d-9268-7ad8298922a7-plugins-config-volume\") pod \"26b8069e-48b2-463d-9268-7ad8298922a7\" (UID: \"26b8069e-48b2-463d-9268-7ad8298922a7\") "
Oct 29 22:40:18.201212 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.200874    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b8069e-48b2-463d-9268-7ad8298922a7-kube-api-access-697jr\" (UniqueName: \"kubernetes.io/projected/26b8069e-48b2-463d-9268-7ad8298922a7-kube-api-access-697jr\") pod \"26b8069e-48b2-463d-9268-7ad8298922a7\" (UID: \"26b8069e-48b2-463d-9268-7ad8298922a7\") "
Oct 29 22:40:18.201265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.201217    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b8069e-48b2-463d-9268-7ad8298922a7-plugins-config-volume" pod "26b8069e-48b2-463d-9268-7ad8298922a7" (UID: "26b8069e-48b2-463d-9268-7ad8298922a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:18.201662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.201638    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b8069e-48b2-463d-9268-7ad8298922a7-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:18.203149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.203114    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b8069e-48b2-463d-9268-7ad8298922a7-kube-api-access-697jr" pod "26b8069e-48b2-463d-9268-7ad8298922a7" (UID: "26b8069e-48b2-463d-9268-7ad8298922a7"). InnerVolumeSpecName "kube-api-access-697jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:18.303235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.303169    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-697jr\" (UniqueName: \"kubernetes.io/projected/26b8069e-48b2-463d-9268-7ad8298922a7-kube-api-access-697jr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:18.715136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.715106    4860 generic.go:356] "Generic (PLEG): container finished" podID="26b8069e-48b2-463d-9268-7ad8298922a7" containerID="fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4" exitCode=0
Oct 29 22:40:18.715282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.715146    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"
Oct 29 22:40:18.715282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.715179    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n" event={"ID":"26b8069e-48b2-463d-9268-7ad8298922a7","Type":"ContainerDied","Data":"fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4"}
Oct 29 22:40:18.715282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.715232    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n" event={"ID":"26b8069e-48b2-463d-9268-7ad8298922a7","Type":"ContainerDied","Data":"b52ad140699e411a4e0a8e32fdfd8d03f1a99912588298dbd92fa52c85f11519"}
Oct 29 22:40:18.715282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.715256    4860 scope.go:122] "RemoveContainer" containerID="fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4"
Oct 29 22:40:18.728138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.728121    4860 scope.go:122] "RemoveContainer" containerID="fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4"
Oct 29 22:40:18.728429 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:40:18.728412    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4\": container with ID starting with fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4 not found: ID does not exist" containerID="fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4"
Oct 29 22:40:18.728486 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.728435    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4"} err="failed to get container status \"fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4\": rpc error: code = NotFound desc = could not find container \"fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4\": container with ID starting with fc76212a6d1236cd602a5c0441a2f0ddd424b31c3678237c4c290da58c9602e4 not found: ID does not exist"
Oct 29 22:40:18.779574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.779550    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"]
Oct 29 22:40:18.793475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:18.793452    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aeda8c-5cc9444875-zqf4n"]
Oct 29 22:40:19.156666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:19.156592    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:19.686081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:19.686050    4860 scope.go:122] "RemoveContainer" containerID="b02471fcc7ca16b0f69f6bc09deaf70a942bac5b3090f83abffdc62f2f0fb7a1"
Oct 29 22:40:19.697576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:19.697548    4860 scope.go:122] "RemoveContainer" containerID="534e047eb87a69810c7fb09e1a0180b043c5a7abc5596e6e0e3afcd608352499"
Oct 29 22:40:19.708628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:19.708607    4860 scope.go:122] "RemoveContainer" containerID="dd1a12f3ba1bdff7bc3bd78cd1e1a540d09052b1402979d304c8ebd7a5ad85de"
Oct 29 22:40:19.718477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:19.718457    4860 scope.go:122] "RemoveContainer" containerID="7a8324ba984eeb3e79c31c6734f246fdfacfb12c11c9283ff88eeb1966eb869b"
Oct 29 22:40:20.083476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:20.083392    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:20.122079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:20.122042    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b8069e-48b2-463d-9268-7ad8298922a7" path="/var/lib/kubelet/pods/26b8069e-48b2-463d-9268-7ad8298922a7/volumes"
Oct 29 22:40:20.211940 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:20.211903    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:20.932346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:20.929420    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:21.054864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:21.054827    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:21.700016 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:21.699984    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:40:22.589078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:22.589042    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:22.863521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:22.863434    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:24.208039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:24.208006    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:26.019015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:26.018981    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:26.144347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:26.143551    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:26.491714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:26.491668    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:27.588274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.588241    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"]
Oct 29 22:40:27.589006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.588990    4860 cpu_manager.go:414] "containerMap: removing container" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy"
Oct 29 22:40:27.589046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.589007    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy"
Oct 29 22:40:27.589079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.589049    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e56007fe-52ad-46e2-845d-681a239a8bf3" containerName="revoke-keys"
Oct 29 22:40:27.589079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.589055    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56007fe-52ad-46e2-845d-681a239a8bf3" containerName="revoke-keys"
Oct 29 22:40:27.589079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.589075    4860 cpu_manager.go:414] "containerMap: removing container" podUID="26b8069e-48b2-463d-9268-7ad8298922a7" containerName="payload-pre-processing"
Oct 29 22:40:27.589186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.589080    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b8069e-48b2-463d-9268-7ad8298922a7" containerName="payload-pre-processing"
Oct 29 22:40:27.589223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.589187    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56007fe-52ad-46e2-845d-681a239a8bf3" containerName="revoke-keys"
Oct 29 22:40:27.589223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.589198    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b8069e-48b2-463d-9268-7ad8298922a7" containerName="payload-pre-processing"
Oct 29 22:40:27.589223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.589205    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy"
Oct 29 22:40:27.589223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.589212    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy"
Oct 29 22:40:27.595921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.595897    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.617413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.617386    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"]
Oct 29 22:40:27.664210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.664186    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.664375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.664229    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsmzn\" (UniqueName: \"kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-kube-api-access-tsmzn\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.664495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.664470    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.664614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.664594    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55380b90-4a75-46b2-ae7b-17f5897ae309-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.664685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.664646    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.664733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.664694    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.664791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.664770    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.664954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.664935    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.665039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.664982    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.764897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.764859    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:27.766135 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766107    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.766248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766164    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55380b90-4a75-46b2-ae7b-17f5897ae309-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.766248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766189    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.766248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766205    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.766248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766224    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.766467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766292    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.766467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766315    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.766467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766397    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.766467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766439    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsmzn\" (UniqueName: \"kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-kube-api-access-tsmzn\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.766691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766668    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.766756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766739    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-certs\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.766836 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766817    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-data\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.767015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.766991    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55380b90-4a75-46b2-ae7b-17f5897ae309-istiod-ca-cert\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.767121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.767024    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-credential-socket\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.768798 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.768772    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-envoy\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.768938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.768921    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-podinfo\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.798051 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.797974    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-token\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.813309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.813262    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsmzn\" (UniqueName: \"kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-kube-api-access-tsmzn\") pod \"e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") " pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:27.916582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:27.916499    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:28.244537 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:40:28.244506    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55380b90_4a75_46b2_ae7b_17f5897ae309.slice/crio-d13d4f452f316a65185dd7d214e53870fbec9c4bd10e3044ce92b1f053d5a488 WatchSource:0}: Error finding container d13d4f452f316a65185dd7d214e53870fbec9c4bd10e3044ce92b1f053d5a488: Status 404 returned error can't find the container with id d13d4f452f316a65185dd7d214e53870fbec9c4bd10e3044ce92b1f053d5a488
Oct 29 22:40:28.247562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:28.247520    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:40:28.247659 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:28.247610    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:40:28.247659 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:28.247651    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:40:28.247728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:28.247655    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"]
Oct 29 22:40:28.834833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:28.834800    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9" event={"ID":"55380b90-4a75-46b2-ae7b-17f5897ae309","Type":"ContainerStarted","Data":"09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807"}
Oct 29 22:40:28.835238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:28.834856    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9" event={"ID":"55380b90-4a75-46b2-ae7b-17f5897ae309","Type":"ContainerStarted","Data":"d13d4f452f316a65185dd7d214e53870fbec9c4bd10e3044ce92b1f053d5a488"}
Oct 29 22:40:28.910087 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:28.910033    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9" podStartSLOduration=1.910018736 podStartE2EDuration="1.910018736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:28.89273626 +0000 UTC m=+1570.917512414" watchObservedRunningTime="2026-10-29 22:40:28.910018736 +0000 UTC m=+1570.934794868"
Oct 29 22:40:28.917339 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:28.917297    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:29.402751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:29.402709    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:29.919588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:29.919551    4860 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:40:29.920000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:29.919627    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9" podUID="55380b90-4a75-46b2-ae7b-17f5897ae309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 29 22:40:30.917435 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:30.917401    4860 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:40:30.917607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:30.917454    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9" podUID="55380b90-4a75-46b2-ae7b-17f5897ae309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 29 22:40:31.292613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:31.292568    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:31.916920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:31.916883    4860 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:40:31.917095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:31.917004    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9" podUID="55380b90-4a75-46b2-ae7b-17f5897ae309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 29 22:40:32.917122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:32.917090    4860 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:40:32.917551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:32.917141    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9" podUID="55380b90-4a75-46b2-ae7b-17f5897ae309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 29 22:40:32.938839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:32.938809    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:33.035421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:33.035379    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"]
Oct 29 22:40:33.046548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:33.046520    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5720f895-nvhnf"]
Oct 29 22:40:33.917384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:33.917349    4860 patch_prober.go:29] interesting pod/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:40:33.917890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:33.917410    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9" podUID="55380b90-4a75-46b2-ae7b-17f5897ae309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Oct 29 22:40:34.118709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:34.118667    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa59d6b-9bec-433d-bf28-19992dc89f36" path="/var/lib/kubelet/pods/4aa59d6b-9bec-433d-bf28-19992dc89f36/volumes"
Oct 29 22:40:34.851871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:34.851833    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:35.505862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:35.505830    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:35.506439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:35.506360    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:35.507233 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:35.507213    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:40:36.549178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:36.549141    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:37.443071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:37.443038    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"]
Oct 29 22:40:37.443254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:37.443233    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" podUID="9bc12952-6dd5-4269-aa22-55b256dddbc1" containerName="istio-proxy" containerID="cri-o://3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe" gracePeriod=30
Oct 29 22:40:37.457876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:37.457843    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"]
Oct 29 22:40:37.458097 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:37.458073    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6" podUID="6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" containerName="istio-proxy" containerID="cri-o://daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df" gracePeriod=30
Oct 29 22:40:37.737143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:37.737066    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"]
Oct 29 22:40:37.737608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:37.737269    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy" containerID="cri-o://5ab721d8b6f631a3947b2e0731cb3d529b2b066a381e98e2566c50fc8dbce903" gracePeriod=30
Oct 29 22:40:38.193065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:38.193021    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:39.780586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:39.780549    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:40.269617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:40.269584    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:41.450560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:41.450520    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:42.764340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.764307    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:40:42.847554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.847527    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc12952-6dd5-4269-aa22-55b256dddbc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc12952-6dd5-4269-aa22-55b256dddbc1-istiod-ca-cert\") pod \"9bc12952-6dd5-4269-aa22-55b256dddbc1\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") "
Oct 29 22:40:42.847708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.847575    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-podinfo\") pod \"9bc12952-6dd5-4269-aa22-55b256dddbc1\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") "
Oct 29 22:40:42.847708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.847611    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-data\") pod \"9bc12952-6dd5-4269-aa22-55b256dddbc1\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") "
Oct 29 22:40:42.847708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.847705    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-token\" (UniqueName: \"kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-token\") pod \"9bc12952-6dd5-4269-aa22-55b256dddbc1\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") "
Oct 29 22:40:42.847902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.847774    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-certs\") pod \"9bc12952-6dd5-4269-aa22-55b256dddbc1\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") "
Oct 29 22:40:42.848058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.848032    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc12952-6dd5-4269-aa22-55b256dddbc1-istiod-ca-cert" pod "9bc12952-6dd5-4269-aa22-55b256dddbc1" (UID: "9bc12952-6dd5-4269-aa22-55b256dddbc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:42.848142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.848034    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-certs" pod "9bc12952-6dd5-4269-aa22-55b256dddbc1" (UID: "9bc12952-6dd5-4269-aa22-55b256dddbc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.848142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.848081    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-envoy\") pod \"9bc12952-6dd5-4269-aa22-55b256dddbc1\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") "
Oct 29 22:40:42.848263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.848208    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-kube-api-access-srbxs\" (UniqueName: \"kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-kube-api-access-srbxs\") pod \"9bc12952-6dd5-4269-aa22-55b256dddbc1\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") "
Oct 29 22:40:42.848525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.848499    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-data" pod "9bc12952-6dd5-4269-aa22-55b256dddbc1" (UID: "9bc12952-6dd5-4269-aa22-55b256dddbc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.848616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.848551    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-credential-socket\") pod \"9bc12952-6dd5-4269-aa22-55b256dddbc1\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") "
Oct 29 22:40:42.848685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.848613    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-socket\") pod \"9bc12952-6dd5-4269-aa22-55b256dddbc1\" (UID: \"9bc12952-6dd5-4269-aa22-55b256dddbc1\") "
Oct 29 22:40:42.848876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.848820    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-credential-socket" pod "9bc12952-6dd5-4269-aa22-55b256dddbc1" (UID: "9bc12952-6dd5-4269-aa22-55b256dddbc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.848954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.848883    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-socket" pod "9bc12952-6dd5-4269-aa22-55b256dddbc1" (UID: "9bc12952-6dd5-4269-aa22-55b256dddbc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.849759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.849737    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.849864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.849764    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.849864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.849779    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.849864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.849792    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.849864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.849808    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc12952-6dd5-4269-aa22-55b256dddbc1-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.850849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.850827    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-podinfo" pod "9bc12952-6dd5-4269-aa22-55b256dddbc1" (UID: "9bc12952-6dd5-4269-aa22-55b256dddbc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:42.851269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.851192    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-envoy" pod "9bc12952-6dd5-4269-aa22-55b256dddbc1" (UID: "9bc12952-6dd5-4269-aa22-55b256dddbc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.851701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.851677    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-token" pod "9bc12952-6dd5-4269-aa22-55b256dddbc1" (UID: "9bc12952-6dd5-4269-aa22-55b256dddbc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:42.851885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.851859    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-kube-api-access-srbxs" pod "9bc12952-6dd5-4269-aa22-55b256dddbc1" (UID: "9bc12952-6dd5-4269-aa22-55b256dddbc1"). InnerVolumeSpecName "kube-api-access-srbxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:42.855657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.855633    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:40:42.950487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.950453    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-podinfo\") pod \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") "
Oct 29 22:40:42.950662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.950605    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-credential-socket\") pod \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") "
Oct 29 22:40:42.950991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.950961    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-credential-socket" pod "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" (UID: "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.951130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.951048    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-kube-api-access-vxs54\" (UniqueName: \"kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-kube-api-access-vxs54\") pod \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") "
Oct 29 22:40:42.951130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.951084    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-certs\") pod \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") "
Oct 29 22:40:42.951130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.951125    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-token\" (UniqueName: \"kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-token\") pod \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") "
Oct 29 22:40:42.951305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.951177    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-data\") pod \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") "
Oct 29 22:40:42.951476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.951370    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istiod-ca-cert\") pod \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") "
Oct 29 22:40:42.951476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.951398    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-envoy\") pod \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") "
Oct 29 22:40:42.951476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.951413    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-certs" pod "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" (UID: "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.951476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.951444    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-socket\") pod \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\" (UID: \"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4\") "
Oct 29 22:40:42.951730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.951707    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-data" pod "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" (UID: "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.952039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.952014    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istiod-ca-cert" pod "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" (UID: "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:42.952039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.952032    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-socket" pod "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" (UID: "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:42.952277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.952243    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.952277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.952272    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.952465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.952286    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.952465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.952299    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.952465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.952312    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srbxs\" (UniqueName: \"kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-kube-api-access-srbxs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.952465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.952339    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.952465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.952353    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.952465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.952367    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.952465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.952379    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc12952-6dd5-4269-aa22-55b256dddbc1-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:42.957669 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.957644    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-podinfo" pod "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" (UID: "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:42.957768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.957711    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-token" pod "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" (UID: "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:42.957858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.957839    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-kube-api-access-vxs54" pod "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" (UID: "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4"). InnerVolumeSpecName "kube-api-access-vxs54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:42.959799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:42.959772    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-envoy" pod "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" (UID: "6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:43.001127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.001040    4860 generic.go:356] "Generic (PLEG): container finished" podID="6a262353-8924-4f7c-94c5-eebde78895aa" containerID="5ab721d8b6f631a3947b2e0731cb3d529b2b066a381e98e2566c50fc8dbce903" exitCode=0
Oct 29 22:40:43.001127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.001100    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" event={"ID":"6a262353-8924-4f7c-94c5-eebde78895aa","Type":"ContainerDied","Data":"5ab721d8b6f631a3947b2e0731cb3d529b2b066a381e98e2566c50fc8dbce903"}
Oct 29 22:40:43.002968 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.002933    4860 generic.go:356] "Generic (PLEG): container finished" podID="9bc12952-6dd5-4269-aa22-55b256dddbc1" containerID="3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe" exitCode=0
Oct 29 22:40:43.003082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.002996    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"
Oct 29 22:40:43.003082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.003027    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" event={"ID":"9bc12952-6dd5-4269-aa22-55b256dddbc1","Type":"ContainerDied","Data":"3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe"}
Oct 29 22:40:43.003082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.003080    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq" event={"ID":"9bc12952-6dd5-4269-aa22-55b256dddbc1","Type":"ContainerDied","Data":"22e46a4261420ae912f25c6b63e1ca057b17efc2a3742016dbc51df18686e8dd"}
Oct 29 22:40:43.003210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.003108    4860 scope.go:122] "RemoveContainer" containerID="3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe"
Oct 29 22:40:43.005204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.005179    4860 generic.go:356] "Generic (PLEG): container finished" podID="6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" containerID="daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df" exitCode=0
Oct 29 22:40:43.005446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.005295    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6" event={"ID":"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4","Type":"ContainerDied","Data":"daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df"}
Oct 29 22:40:43.005446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.005346    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6" event={"ID":"6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4","Type":"ContainerDied","Data":"ad42950bc3d6bd0e0e18838240e4eb7b54179c83040a63b07bbc397370aa4a04"}
Oct 29 22:40:43.005446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.005412    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"
Oct 29 22:40:43.017934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.017883    4860 scope.go:122] "RemoveContainer" containerID="3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe"
Oct 29 22:40:43.018234 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:40:43.018215    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe\": container with ID starting with 3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe not found: ID does not exist" containerID="3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe"
Oct 29 22:40:43.018313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.018240    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe"} err="failed to get container status \"3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe\": rpc error: code = NotFound desc = could not find container \"3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe\": container with ID starting with 3d0b06bcd3d98fceae4c47ae36e5d0b091805366e790e162f691ffe8657062fe not found: ID does not exist"
Oct 29 22:40:43.018313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.018257    4860 scope.go:122] "RemoveContainer" containerID="daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df"
Oct 29 22:40:43.031477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.031428    4860 scope.go:122] "RemoveContainer" containerID="daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df"
Oct 29 22:40:43.031761 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:40:43.031743    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df\": container with ID starting with daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df not found: ID does not exist" containerID="daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df"
Oct 29 22:40:43.031837 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.031771    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df"} err="failed to get container status \"daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df\": rpc error: code = NotFound desc = could not find container \"daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df\": container with ID starting with daef88e22ae67e102026d75de8ddd171d54a9213dc9e55188588db2b9e4c56df not found: ID does not exist"
Oct 29 22:40:43.054581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.054555    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.054692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.054584    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxs54\" (UniqueName: \"kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-kube-api-access-vxs54\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.054692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.054599    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.054692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.054612    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.055461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.055435    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:43.059742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.059721    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:40:43.111191 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.111152    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"]
Oct 29 22:40:43.120955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.120926    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-cwggq"]
Oct 29 22:40:43.156776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.156606    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-kube-api-access-chzb5\" (UniqueName: \"kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-kube-api-access-chzb5\") pod \"6a262353-8924-4f7c-94c5-eebde78895aa\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") "
Oct 29 22:40:43.156776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.156751    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-data\") pod \"6a262353-8924-4f7c-94c5-eebde78895aa\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") "
Oct 29 22:40:43.157035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.156868    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-socket\") pod \"6a262353-8924-4f7c-94c5-eebde78895aa\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") "
Oct 29 22:40:43.157035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.156912    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-certs\") pod \"6a262353-8924-4f7c-94c5-eebde78895aa\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") "
Oct 29 22:40:43.157035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.156993    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-istio-token\" (UniqueName: \"kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-istio-token\") pod \"6a262353-8924-4f7c-94c5-eebde78895aa\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") "
Oct 29 22:40:43.157206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.157061    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a262353-8924-4f7c-94c5-eebde78895aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a262353-8924-4f7c-94c5-eebde78895aa-istiod-ca-cert\") pod \"6a262353-8924-4f7c-94c5-eebde78895aa\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") "
Oct 29 22:40:43.157206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.157116    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-credential-socket\") pod \"6a262353-8924-4f7c-94c5-eebde78895aa\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") "
Oct 29 22:40:43.157206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.157160    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-envoy\") pod \"6a262353-8924-4f7c-94c5-eebde78895aa\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") "
Oct 29 22:40:43.157387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.157374    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a262353-8924-4f7c-94c5-eebde78895aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a262353-8924-4f7c-94c5-eebde78895aa-istio-podinfo\") pod \"6a262353-8924-4f7c-94c5-eebde78895aa\" (UID: \"6a262353-8924-4f7c-94c5-eebde78895aa\") "
Oct 29 22:40:43.160865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.160830    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-socket" pod "6a262353-8924-4f7c-94c5-eebde78895aa" (UID: "6a262353-8924-4f7c-94c5-eebde78895aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:43.161019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.160851    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a262353-8924-4f7c-94c5-eebde78895aa-istio-podinfo" pod "6a262353-8924-4f7c-94c5-eebde78895aa" (UID: "6a262353-8924-4f7c-94c5-eebde78895aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:40:43.161019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.160925    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-certs" pod "6a262353-8924-4f7c-94c5-eebde78895aa" (UID: "6a262353-8924-4f7c-94c5-eebde78895aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:43.161155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.161120    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-data" pod "6a262353-8924-4f7c-94c5-eebde78895aa" (UID: "6a262353-8924-4f7c-94c5-eebde78895aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:43.161155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.161129    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a262353-8924-4f7c-94c5-eebde78895aa-istiod-ca-cert" pod "6a262353-8924-4f7c-94c5-eebde78895aa" (UID: "6a262353-8924-4f7c-94c5-eebde78895aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:40:43.163368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.161443    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-credential-socket" pod "6a262353-8924-4f7c-94c5-eebde78895aa" (UID: "6a262353-8924-4f7c-94c5-eebde78895aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:43.163368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.161546    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-kube-api-access-chzb5" pod "6a262353-8924-4f7c-94c5-eebde78895aa" (UID: "6a262353-8924-4f7c-94c5-eebde78895aa"). InnerVolumeSpecName "kube-api-access-chzb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:43.163368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.162777    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-envoy" pod "6a262353-8924-4f7c-94c5-eebde78895aa" (UID: "6a262353-8924-4f7c-94c5-eebde78895aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:40:43.169979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.169946    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-istio-token" pod "6a262353-8924-4f7c-94c5-eebde78895aa" (UID: "6a262353-8924-4f7c-94c5-eebde78895aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:40:43.191383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.189751    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"]
Oct 29 22:40:43.198750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.198726    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-8wvv6"]
Oct 29 22:40:43.259530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.259458    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a262353-8924-4f7c-94c5-eebde78895aa-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.259530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.259484    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chzb5\" (UniqueName: \"kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-kube-api-access-chzb5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.259530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.259494    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.259530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.259504    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.259530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.259512    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.259530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.259520    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a262353-8924-4f7c-94c5-eebde78895aa-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.259530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.259527    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a262353-8924-4f7c-94c5-eebde78895aa-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.259530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.259535    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:43.259912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:43.259543    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a262353-8924-4f7c-94c5-eebde78895aa-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:40:44.015042 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:44.015014    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"
Oct 29 22:40:44.015497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:44.015037    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg" event={"ID":"6a262353-8924-4f7c-94c5-eebde78895aa","Type":"ContainerDied","Data":"f3e2ff7edb3800222d469730a6c6c480efb64775f35cc2edd39ccc2b54fd6ea8"}
Oct 29 22:40:44.015497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:44.015125    4860 scope.go:122] "RemoveContainer" containerID="5ab721d8b6f631a3947b2e0731cb3d529b2b066a381e98e2566c50fc8dbce903"
Oct 29 22:40:44.074939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:44.074906    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"]
Oct 29 22:40:44.087500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:44.087475    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-8n7fg"]
Oct 29 22:40:44.113632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:44.113607    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" path="/var/lib/kubelet/pods/6a262353-8924-4f7c-94c5-eebde78895aa/volumes"
Oct 29 22:40:44.114111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:44.114096    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" path="/var/lib/kubelet/pods/6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4/volumes"
Oct 29 22:40:44.114581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:44.114567    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc12952-6dd5-4269-aa22-55b256dddbc1" path="/var/lib/kubelet/pods/9bc12952-6dd5-4269-aa22-55b256dddbc1/volumes"
Oct 29 22:40:45.019309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:45.019276    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:46.781052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:46.781023    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:48.058406 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:40:48.058370    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:40:48.663755 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:48.663708    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:49.583160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:49.583125    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:50.520687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:50.520644    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:52.044540 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:52.044508    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:52.324751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:52.324663    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:54.380544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:54.380503    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:56.107933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:56.107882    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:57.032946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:57.032897    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:58.390964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:58.390918    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:40:59.689421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:40:59.689377    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:00.086286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:00.086201    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:00.336513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:00.336419    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:01.704133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:01.704096    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"]
Oct 29 22:41:01.704649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:01.704363    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd" podUID="6a154999-c757-467d-b8be-78e122dea6e6" containerName="maas-api" containerID="cri-o://b1f7e9c88dde03684d6d703dae779a1518e56ed22302cab59aefd2515d7c880e" gracePeriod=30
Oct 29 22:41:01.794108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:01.794072    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:02.260642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.260607    4860 generic.go:356] "Generic (PLEG): container finished" podID="6a154999-c757-467d-b8be-78e122dea6e6" containerID="b1f7e9c88dde03684d6d703dae779a1518e56ed22302cab59aefd2515d7c880e" exitCode=0
Oct 29 22:41:02.260905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.260662    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd" event={"ID":"6a154999-c757-467d-b8be-78e122dea6e6","Type":"ContainerDied","Data":"b1f7e9c88dde03684d6d703dae779a1518e56ed22302cab59aefd2515d7c880e"}
Oct 29 22:41:02.261055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.261042    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd" event={"ID":"6a154999-c757-467d-b8be-78e122dea6e6","Type":"ContainerDied","Data":"d0db2dc4d50c3503573560eb305b7847b7862c07fa585ab1482ddafcb6931be3"}
Oct 29 22:41:02.261143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.261134    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0db2dc4d50c3503573560eb305b7847b7862c07fa585ab1482ddafcb6931be3"
Oct 29 22:41:02.264431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.264413    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:41:02.336918 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.336881    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a154999-c757-467d-b8be-78e122dea6e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a154999-c757-467d-b8be-78e122dea6e6-openshift-service-ca\") pod \"6a154999-c757-467d-b8be-78e122dea6e6\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") "
Oct 29 22:41:02.338407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.337202    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls\") pod \"6a154999-c757-467d-b8be-78e122dea6e6\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") "
Oct 29 22:41:02.338407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.337263    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-metrics-certs\") pod \"6a154999-c757-467d-b8be-78e122dea6e6\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") "
Oct 29 22:41:02.338407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.337296    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a154999-c757-467d-b8be-78e122dea6e6-kube-api-access-llrll\" (UniqueName: \"kubernetes.io/projected/6a154999-c757-467d-b8be-78e122dea6e6-kube-api-access-llrll\") pod \"6a154999-c757-467d-b8be-78e122dea6e6\" (UID: \"6a154999-c757-467d-b8be-78e122dea6e6\") "
Oct 29 22:41:02.338407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.338363    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a154999-c757-467d-b8be-78e122dea6e6-openshift-service-ca" pod "6a154999-c757-467d-b8be-78e122dea6e6" (UID: "6a154999-c757-467d-b8be-78e122dea6e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:02.340500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.339816    4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a154999-c757-467d-b8be-78e122dea6e6-openshift-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:02.340784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.340758    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-metrics-certs" pod "6a154999-c757-467d-b8be-78e122dea6e6" (UID: "6a154999-c757-467d-b8be-78e122dea6e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:41:02.347750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.347713    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls" pod "6a154999-c757-467d-b8be-78e122dea6e6" (UID: "6a154999-c757-467d-b8be-78e122dea6e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:41:02.421434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.420638    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a154999-c757-467d-b8be-78e122dea6e6-kube-api-access-llrll" pod "6a154999-c757-467d-b8be-78e122dea6e6" (UID: "6a154999-c757-467d-b8be-78e122dea6e6"). InnerVolumeSpecName "kube-api-access-llrll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:02.421434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.421290    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:02.443199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.441266    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-maas-api-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:02.443199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.441292    4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a154999-c757-467d-b8be-78e122dea6e6-metrics-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:02.443199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.441301    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llrll\" (UniqueName: \"kubernetes.io/projected/6a154999-c757-467d-b8be-78e122dea6e6-kube-api-access-llrll\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:02.547472 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.547386    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:02.551565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.551537    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"]
Oct 29 22:41:02.559118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:02.559086    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aeda8c-wppfw"]
Oct 29 22:41:03.063525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:03.063486    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:03.272863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:03.272833    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"
Oct 29 22:41:03.346116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:03.345710    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"]
Oct 29 22:41:03.351747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:03.351723    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aeda8c-7b74874f79-h66jd"]
Oct 29 22:41:04.116882 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:04.116839    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a154999-c757-467d-b8be-78e122dea6e6" path="/var/lib/kubelet/pods/6a154999-c757-467d-b8be-78e122dea6e6/volumes"
Oct 29 22:41:04.117708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:04.117687    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d673cfd-c7ec-4f54-a96e-07904b39ab5d" path="/var/lib/kubelet/pods/9d673cfd-c7ec-4f54-a96e-07904b39ab5d/volumes"
Oct 29 22:41:04.314695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:04.314656    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:05.300499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:05.300456    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:41:05.300892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:05.300513    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:41:05.910137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:05.910106    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:06.659421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:06.659383    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:07.312650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:07.312621    4860 generic.go:356] "Generic (PLEG): container finished" podID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerID="de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b" exitCode=0
Oct 29 22:41:07.312835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:07.312696    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" event={"ID":"a3afafb7-388b-4114-82a8-adacbbb15f5d","Type":"ContainerDied","Data":"de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b"}
Oct 29 22:41:07.313040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:07.313025    4860 scope.go:122] "RemoveContainer" containerID="de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b"
Oct 29 22:41:07.314777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:07.314750    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:07.314849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:07.314818    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:07.314887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:07.314851    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:07.561623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:07.561589    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:08.323540 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:08.323508    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" event={"ID":"a3afafb7-388b-4114-82a8-adacbbb15f5d","Type":"ContainerStarted","Data":"725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729"}
Oct 29 22:41:08.911318 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:08.911281    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:41:08.911548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:08.911510    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 29 22:41:08.911632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:08.911565    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 29 22:41:09.148343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:09.148293    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:09.347403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:09.347318    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:09.515064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:09.515026    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"]
Oct 29 22:41:09.515302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:09.515271    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy" containerID="cri-o://62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d" gracePeriod=30
Oct 29 22:41:09.559767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:09.559742    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:09.911916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:09.911883    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 29 22:41:09.912107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:09.911935    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 29 22:41:10.392464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.392411    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:10.804478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.804442    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"]
Oct 29 22:41:10.805662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.805629    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy"
Oct 29 22:41:10.805662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.805657    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy"
Oct 29 22:41:10.806642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.805697    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9bc12952-6dd5-4269-aa22-55b256dddbc1" containerName="istio-proxy"
Oct 29 22:41:10.806642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.805705    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc12952-6dd5-4269-aa22-55b256dddbc1" containerName="istio-proxy"
Oct 29 22:41:10.806642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.805733    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6a154999-c757-467d-b8be-78e122dea6e6" containerName="maas-api"
Oct 29 22:41:10.806642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.805741    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a154999-c757-467d-b8be-78e122dea6e6" containerName="maas-api"
Oct 29 22:41:10.806642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.805764    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" containerName="istio-proxy"
Oct 29 22:41:10.806642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.805771    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" containerName="istio-proxy"
Oct 29 22:41:10.806642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.805793    4860 cpu_manager.go:414] "containerMap: removing container" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy"
Oct 29 22:41:10.806642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.805799    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551bb653-c966-43cc-8f02-007606849b1f" containerName="istio-proxy"
Oct 29 22:41:10.806642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.806038    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a262353-8924-4f7c-94c5-eebde78895aa" containerName="istio-proxy"
Oct 29 22:41:10.806642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.806058    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc12952-6dd5-4269-aa22-55b256dddbc1" containerName="istio-proxy"
Oct 29 22:41:10.806642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.806071    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a154999-c757-467d-b8be-78e122dea6e6" containerName="maas-api"
Oct 29 22:41:10.806642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.806081    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd673ec-820a-467b-9ccf-e2f5e4a0fbf4" containerName="istio-proxy"
Oct 29 22:41:10.810017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.809997    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:10.815041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.815015    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"]
Oct 29 22:41:10.826454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.826431    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fee294-openshift-default-dockercfg-jk6b9\""
Oct 29 22:41:10.830038 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.830016    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:10.936406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.924712    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 29 22:41:10.936406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.924764    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 29 22:41:10.983912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:10.983874    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:11.023348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.020292    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-socket\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.067417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.030510    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-envoy\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.067417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.030575    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-credential-socket\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.067417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.030788    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh6mp\" (UniqueName: \"kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-kube-api-access-zh6mp\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.067417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.030858    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-certs\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.067417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.030964    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-data\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.067417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.031034    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istiod-ca-cert\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.067417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.031069    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-token\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.067417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.031100    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-podinfo\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.144255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.142967    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh6mp\" (UniqueName: \"kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-kube-api-access-zh6mp\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.144255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.143047    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-certs\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.144255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.143123    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-data\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.144255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.143179    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istiod-ca-cert\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.144255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.143210    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-token\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.144255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.143240    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-podinfo\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.144255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.143268    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:11.144255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.143371    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-socket\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.144255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.143412    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-envoy\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.144255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.143439    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-credential-socket\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.144255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.143656    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-data\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.145106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.144444    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-socket\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.145106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.144526    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-credential-socket\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.145106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.144903    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-certs\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.146117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.146069    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istiod-ca-cert\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.151007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.150962    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-podinfo\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.162000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.161979    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-envoy\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.167988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.167597    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-token\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.174649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.174595    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh6mp\" (UniqueName: \"kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-kube-api-access-zh6mp\") pod \"e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") " pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.428816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.428727    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:11.796476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.796439    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"]
Oct 29 22:41:11.799651 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:41:11.799624    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bb03a1_504e_4c1e_8df1_d6af23cd10ed.slice/crio-2a6ec0ccda9a38f21dfd82367444b206faee149c2cf0ee2264aad4df669053c1 WatchSource:0}: Error finding container 2a6ec0ccda9a38f21dfd82367444b206faee149c2cf0ee2264aad4df669053c1: Status 404 returned error can't find the container with id 2a6ec0ccda9a38f21dfd82367444b206faee149c2cf0ee2264aad4df669053c1
Oct 29 22:41:11.802520 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.802484    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:11.802613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.802553    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:11.802613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.802586    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:11.911855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.911823    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 29 22:41:11.911975 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:11.911883    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 29 22:41:12.252458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:12.252423    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:12.380790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:12.372571    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" event={"ID":"98bb03a1-504e-4c1e-8df1-d6af23cd10ed","Type":"ContainerStarted","Data":"a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de"}
Oct 29 22:41:12.380790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:12.372640    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" event={"ID":"98bb03a1-504e-4c1e-8df1-d6af23cd10ed","Type":"ContainerStarted","Data":"2a6ec0ccda9a38f21dfd82367444b206faee149c2cf0ee2264aad4df669053c1"}
Oct 29 22:41:12.429583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:12.429551    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:12.438764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:12.438682    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" podStartSLOduration=2.438665336 podStartE2EDuration="2.438665336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:12.42478305 +0000 UTC m=+1614.449559230" watchObservedRunningTime="2026-10-29 22:41:12.438665336 +0000 UTC m=+1614.463441482"
Oct 29 22:41:12.439170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:12.439148    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:12.450985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:12.450944    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:12.911962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:12.911928    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 29 22:41:12.912146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:12.911987    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 29 22:41:13.127320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:13.127285    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:13.234431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:13.234392    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"]
Oct 29 22:41:13.240083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:13.240049    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39322f-9z97s"]
Oct 29 22:41:13.331108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:13.329873    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:13.437527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:13.437477    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:13.438775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:13.438751    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:41:13.445171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:13.445140    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:13.792349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:13.790168    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:13.911599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:13.911566    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 29 22:41:13.911809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:13.911622    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 29 22:41:14.113919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.113840    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52dec074-2dee-4aa2-962e-66be90f779b2" path="/var/lib/kubelet/pods/52dec074-2dee-4aa2-962e-66be90f779b2/volumes"
Oct 29 22:41:14.819592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.819564    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:41:14.831726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.831702    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-podinfo\") pod \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") "
Oct 29 22:41:14.831848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.831745    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-socket\") pod \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") "
Oct 29 22:41:14.831848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.831813    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-credential-socket\") pod \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") "
Oct 29 22:41:14.831953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.831856    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-certs\") pod \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") "
Oct 29 22:41:14.831953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.831872    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-envoy\") pod \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") "
Oct 29 22:41:14.831953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.831899    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-data\") pod \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") "
Oct 29 22:41:14.831953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.831925    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-kube-api-access-vm6vw\" (UniqueName: \"kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-kube-api-access-vm6vw\") pod \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") "
Oct 29 22:41:14.832248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.832044    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-token\" (UniqueName: \"kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-token\") pod \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") "
Oct 29 22:41:14.832248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.832090    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istiod-ca-cert\") pod \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\" (UID: \"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7\") "
Oct 29 22:41:14.832391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.832084    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-socket" pod "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" (UID: "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:14.832453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.832233    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-data" pod "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" (UID: "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:14.832511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.832250    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-certs" pod "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" (UID: "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:14.832511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.832455    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-credential-socket" pod "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" (UID: "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:14.832856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.832828    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istiod-ca-cert" pod "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" (UID: "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:14.833308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.833284    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:14.833480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.833313    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:14.833480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.833344    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:14.833480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.833356    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:14.833480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.833368    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:14.834253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.834227    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-podinfo" pod "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" (UID: "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:14.834417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.834393    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-envoy" pod "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" (UID: "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:14.834585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.834564    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-token" pod "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" (UID: "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:14.834915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.834894    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-kube-api-access-vm6vw" pod "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" (UID: "3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7"). InnerVolumeSpecName "kube-api-access-vm6vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:14.917821 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.917784    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:41:14.919508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.919241    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:41:14.924644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.924593    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:41:14.935346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.934373    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:14.935346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.935094    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:14.935346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.935113    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:14.935346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.935126    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm6vw\" (UniqueName: \"kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-kube-api-access-vm6vw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:14.935346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:14.935150    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:15.078895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.078794    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:15.459060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.459021    4860 generic.go:356] "Generic (PLEG): container finished" podID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerID="62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d" exitCode=0
Oct 29 22:41:15.459244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.459114    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"
Oct 29 22:41:15.459244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.459146    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" event={"ID":"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7","Type":"ContainerDied","Data":"62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d"}
Oct 29 22:41:15.459244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.459210    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv" event={"ID":"3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7","Type":"ContainerDied","Data":"12fd6c8798fc8d3fd591bb62fadf8951241f42a9071408bf0ea15c75bdf11d0e"}
Oct 29 22:41:15.459470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.459244    4860 scope.go:122] "RemoveContainer" containerID="62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d"
Oct 29 22:41:15.475169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.475147    4860 scope.go:122] "RemoveContainer" containerID="8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a"
Oct 29 22:41:15.487098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.487075    4860 scope.go:122] "RemoveContainer" containerID="62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d"
Oct 29 22:41:15.487493 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:41:15.487464    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d\": container with ID starting with 62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d not found: ID does not exist" containerID="62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d"
Oct 29 22:41:15.487628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.487503    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d"} err="failed to get container status \"62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d\": rpc error: code = NotFound desc = could not find container \"62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d\": container with ID starting with 62ca4c00e908b45c4dc0d8cce9c2dc954a1d1e1f5e9b32eed15762b1ee6d816d not found: ID does not exist"
Oct 29 22:41:15.487628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.487534    4860 scope.go:122] "RemoveContainer" containerID="8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a"
Oct 29 22:41:15.487827 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:41:15.487808    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a\": container with ID starting with 8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a not found: ID does not exist" containerID="8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a"
Oct 29 22:41:15.487890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.487837    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a"} err="failed to get container status \"8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a\": rpc error: code = NotFound desc = could not find container \"8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a\": container with ID starting with 8b7ac98ce2cfc25d58b6ba12cc255ec545349ddd11135392f21d066d4decea9a not found: ID does not exist"
Oct 29 22:41:15.533712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.533685    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"]
Oct 29 22:41:15.547201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.547170    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-aeda8c-openshift-default-57d7447bc-6nmrv"]
Oct 29 22:41:15.741108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.741030    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:15.870014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.869987    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:15.967040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.967010    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"]
Oct 29 22:41:15.977785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.977754    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3d1777-ng489"]
Oct 29 22:41:15.998607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:15.998548    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:16.117036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.116993    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" path="/var/lib/kubelet/pods/3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7/volumes"
Oct 29 22:41:16.117704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.117682    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56007fe-52ad-46e2-845d-681a239a8bf3" path="/var/lib/kubelet/pods/e56007fe-52ad-46e2-845d-681a239a8bf3/volumes"
Oct 29 22:41:16.465769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.465742    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:16.674885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.674801    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:16.800705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.800619    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:16.825451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.825426    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"]
Oct 29 22:41:16.826193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.826165    4860 cpu_manager.go:414] "containerMap: removing container" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy"
Oct 29 22:41:16.826193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.826186    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy"
Oct 29 22:41:16.826367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.826221    4860 cpu_manager.go:414] "containerMap: removing container" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy"
Oct 29 22:41:16.826367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.826227    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy"
Oct 29 22:41:16.826448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.826411    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy"
Oct 29 22:41:16.829882 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.829864    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"
Oct 29 22:41:16.847867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.847845    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fee294-dockercfg-l4jqd\""
Oct 29 22:41:16.847989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.847889    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fee294\""
Oct 29 22:41:16.868488 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.868457    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e35c643-1248-47c8-8acd-1a4991f0a395-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t\" (UID: \"1e35c643-1248-47c8-8acd-1a4991f0a395\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"
Oct 29 22:41:16.868735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.868705    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bksl\" (UniqueName: \"kubernetes.io/projected/1e35c643-1248-47c8-8acd-1a4991f0a395-kube-api-access-9bksl\") pod \"payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t\" (UID: \"1e35c643-1248-47c8-8acd-1a4991f0a395\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"
Oct 29 22:41:16.869123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:16.869102    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"]
Oct 29 22:41:17.027347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.025444    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"]
Oct 29 22:41:17.049868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.049827    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e35c643-1248-47c8-8acd-1a4991f0a395-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t\" (UID: \"1e35c643-1248-47c8-8acd-1a4991f0a395\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"
Oct 29 22:41:17.050062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.049913    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bksl\" (UniqueName: \"kubernetes.io/projected/1e35c643-1248-47c8-8acd-1a4991f0a395-kube-api-access-9bksl\") pod \"payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t\" (UID: \"1e35c643-1248-47c8-8acd-1a4991f0a395\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"
Oct 29 22:41:17.053139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.053062    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e35c643-1248-47c8-8acd-1a4991f0a395-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t\" (UID: \"1e35c643-1248-47c8-8acd-1a4991f0a395\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"
Oct 29 22:41:17.064692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.064661    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d794f4f-1519-4aa9-8cd5-dc6016b7bdb7" containerName="istio-proxy"
Oct 29 22:41:17.069978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.069947    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"]
Oct 29 22:41:17.070114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.070059    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"
Oct 29 22:41:17.087887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.087864    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bksl\" (UniqueName: \"kubernetes.io/projected/1e35c643-1248-47c8-8acd-1a4991f0a395-kube-api-access-9bksl\") pod \"payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t\" (UID: \"1e35c643-1248-47c8-8acd-1a4991f0a395\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"
Oct 29 22:41:17.143818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.143786    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"
Oct 29 22:41:17.257675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.257198    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6xtw\" (UniqueName: \"kubernetes.io/projected/70d957f1-c5a8-4993-a3af-2e35b2934cc4-kube-api-access-n6xtw\") pod \"payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx\" (UID: \"70d957f1-c5a8-4993-a3af-2e35b2934cc4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"
Oct 29 22:41:17.257675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.257350    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70d957f1-c5a8-4993-a3af-2e35b2934cc4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx\" (UID: \"70d957f1-c5a8-4993-a3af-2e35b2934cc4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"
Oct 29 22:41:17.359528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.359458    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6xtw\" (UniqueName: \"kubernetes.io/projected/70d957f1-c5a8-4993-a3af-2e35b2934cc4-kube-api-access-n6xtw\") pod \"payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx\" (UID: \"70d957f1-c5a8-4993-a3af-2e35b2934cc4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"
Oct 29 22:41:17.359712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.359537    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70d957f1-c5a8-4993-a3af-2e35b2934cc4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx\" (UID: \"70d957f1-c5a8-4993-a3af-2e35b2934cc4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"
Oct 29 22:41:17.360277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.360250    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70d957f1-c5a8-4993-a3af-2e35b2934cc4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx\" (UID: \"70d957f1-c5a8-4993-a3af-2e35b2934cc4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"
Oct 29 22:41:17.380939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.380918    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6xtw\" (UniqueName: \"kubernetes.io/projected/70d957f1-c5a8-4993-a3af-2e35b2934cc4-kube-api-access-n6xtw\") pod \"payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx\" (UID: \"70d957f1-c5a8-4993-a3af-2e35b2934cc4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"
Oct 29 22:41:17.387703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.387677    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"
Oct 29 22:41:17.476834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.475776    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"]
Oct 29 22:41:17.691369 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:41:17.691338    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d957f1_c5a8_4993_a3af_2e35b2934cc4.slice/crio-6fbd78474f575ac65d790907241eea125503839bc95f6be33f22da7135716dc8 WatchSource:0}: Error finding container 6fbd78474f575ac65d790907241eea125503839bc95f6be33f22da7135716dc8: Status 404 returned error can't find the container with id 6fbd78474f575ac65d790907241eea125503839bc95f6be33f22da7135716dc8
Oct 29 22:41:17.694932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.694903    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"]
Oct 29 22:41:17.960718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:17.960631    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:18.060982 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:41:18.060940    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:41:18.496302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:18.496268    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx" event={"ID":"70d957f1-c5a8-4993-a3af-2e35b2934cc4","Type":"ContainerStarted","Data":"a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce"}
Oct 29 22:41:18.496535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:18.496337    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"
Oct 29 22:41:18.496535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:18.496364    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx" event={"ID":"70d957f1-c5a8-4993-a3af-2e35b2934cc4","Type":"ContainerStarted","Data":"6fbd78474f575ac65d790907241eea125503839bc95f6be33f22da7135716dc8"}
Oct 29 22:41:18.497974 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:18.497954    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t" event={"ID":"1e35c643-1248-47c8-8acd-1a4991f0a395","Type":"ContainerStarted","Data":"1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652"}
Oct 29 22:41:18.498063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:18.497990    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t" event={"ID":"1e35c643-1248-47c8-8acd-1a4991f0a395","Type":"ContainerStarted","Data":"8ad703d20d0542b9385f64bbaeedbab4a72fd2c14f6985e5f145e406e0e30a8b"}
Oct 29 22:41:18.498063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:18.498035    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"
Oct 29 22:41:18.538972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:18.538920    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx" podStartSLOduration=2.538906639 podStartE2EDuration="2.538906639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:18.537034898 +0000 UTC m=+1620.561811045" watchObservedRunningTime="2026-10-29 22:41:18.538906639 +0000 UTC m=+1620.563682773"
Oct 29 22:41:18.579466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:18.579425    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t" podStartSLOduration=2.579412349 podStartE2EDuration="2.579412349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:18.577013903 +0000 UTC m=+1620.601790054" watchObservedRunningTime="2026-10-29 22:41:18.579412349 +0000 UTC m=+1620.604188480"
Oct 29 22:41:19.029167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:19.029138    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"]
Oct 29 22:41:19.042450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:19.042423    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9684486f-xpv5k"]
Oct 29 22:41:19.636853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:19.636819    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:19.819703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:19.819672    4860 scope.go:122] "RemoveContainer" containerID="09f9a8b44e814d82ee91473f2debe7e28499b08d1098a0409b5cf6bdb028a82d"
Oct 29 22:41:19.833563 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:19.833542    4860 scope.go:122] "RemoveContainer" containerID="8d897306a3e6fe9483a5efc3a11c4a52b8285a321961f1bb9e4af0fbeb693ed3"
Oct 29 22:41:19.844044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:19.844024    4860 scope.go:122] "RemoveContainer" containerID="8c0cd93050cc63b66c758f964fc6863a2d6339d107c03287062bdc801410484b"
Oct 29 22:41:19.855234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:19.855215    4860 scope.go:122] "RemoveContainer" containerID="f3ad53237e76c533ff4efb4e5c3e8354c3b8b8205d6c05688cd976ddff024b99"
Oct 29 22:41:20.126025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:20.125994    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625caf72-c0df-499f-b684-ff559d204460" path="/var/lib/kubelet/pods/625caf72-c0df-499f-b684-ff559d204460/volumes"
Oct 29 22:41:20.214980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:20.214928    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:20.336578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:20.336545    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:20.508224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:20.508140    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"]
Oct 29 22:41:20.510310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:20.510274    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy" containerID="cri-o://ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d" gracePeriod=30
Oct 29 22:41:20.552385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:20.552355    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:21.231419 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.231380    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:21.360498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.360464    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:21.719226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.719188    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"]
Oct 29 22:41:21.735143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.735116    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.748526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.748503    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"]
Oct 29 22:41:21.755211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.755185    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a5af18-openshift-default-dockercfg-jxpqf\""
Oct 29 22:41:21.786175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.786147    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:21.891345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.868359    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-envoy\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.891345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.868441    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-credential-socket\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.891345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.868477    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-data\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.891345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.868541    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-podinfo\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.891345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.868614    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-certs\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.891345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.868679    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb4nd\" (UniqueName: \"kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-kube-api-access-jb4nd\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.891345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.868743    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-socket\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.891345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.868809    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-token\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.891345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.868976    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307076b4-eceb-48f7-8ca6-5e11f666c439-istiod-ca-cert\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.971412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.970428    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307076b4-eceb-48f7-8ca6-5e11f666c439-istiod-ca-cert\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.971412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.970565    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-envoy\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.971412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.970602    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-credential-socket\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.971412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.970628    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-data\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.971412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.970685    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-podinfo\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.971412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.970751    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-certs\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.971412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.970812    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb4nd\" (UniqueName: \"kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-kube-api-access-jb4nd\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.971412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.970873    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-socket\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.971412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.970947    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-token\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:21.990407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.972256    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307076b4-eceb-48f7-8ca6-5e11f666c439-istiod-ca-cert\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:22.000350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.990748    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-credential-socket\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:22.000350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:21.991017    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-data\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:22.034729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.034687    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-certs\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:22.046543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.035886    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-socket\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:22.046543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.036079    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-envoy\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:22.046543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.041376    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:22.060538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.059920    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-podinfo\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:22.069702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.067028    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb4nd\" (UniqueName: \"kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-kube-api-access-jb4nd\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:22.088958 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.074979    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-token\") pod \"e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") " pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:22.174212 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.174174    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:22.352181 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.352092    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:22.684283 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:41:22.684253    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307076b4_eceb_48f7_8ca6_5e11f666c439.slice/crio-c77adf228672dfe81fd8f66cd46a7cc96f6db20f5e474b99d0f83542c72a95aa WatchSource:0}: Error finding container c77adf228672dfe81fd8f66cd46a7cc96f6db20f5e474b99d0f83542c72a95aa: Status 404 returned error can't find the container with id c77adf228672dfe81fd8f66cd46a7cc96f6db20f5e474b99d0f83542c72a95aa
Oct 29 22:41:22.686560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.686537    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"]
Oct 29 22:41:22.687139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.687105    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:22.687222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.687182    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:22.687287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:22.687227    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:23.285952 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:23.285912    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:23.558222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:23.558122    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" event={"ID":"307076b4-eceb-48f7-8ca6-5e11f666c439","Type":"ContainerStarted","Data":"a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe"}
Oct 29 22:41:23.558222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:23.558188    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" event={"ID":"307076b4-eceb-48f7-8ca6-5e11f666c439","Type":"ContainerStarted","Data":"c77adf228672dfe81fd8f66cd46a7cc96f6db20f5e474b99d0f83542c72a95aa"}
Oct 29 22:41:23.636549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:23.636504    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" podStartSLOduration=2.636490418 podStartE2EDuration="2.636490418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:23.620765853 +0000 UTC m=+1625.645542003" watchObservedRunningTime="2026-10-29 22:41:23.636490418 +0000 UTC m=+1625.661266549"
Oct 29 22:41:24.353041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:24.353009    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:24.359112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:24.359087    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:24.569822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:24.569782    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:24.570775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:24.570756    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:41:24.891907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:24.891857    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:25.038030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:25.037990    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:25.212566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:25.212537    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:25.446620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:25.446580    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:25.585545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:25.585516    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:25.738757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:25.738732    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"]
Oct 29 22:41:25.739500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:25.739204    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy" containerID="cri-o://720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35" gracePeriod=30
Oct 29 22:41:25.756228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:25.754263    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"]
Oct 29 22:41:25.756228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:25.754518    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9" podUID="55380b90-4a75-46b2-ae7b-17f5897ae309" containerName="istio-proxy" containerID="cri-o://09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807" gracePeriod=30
Oct 29 22:41:25.799711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:25.799631    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:25.961728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:25.958425    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:25.965852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:25.965827    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:41:26.066518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.065979    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-certs\") pod \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") "
Oct 29 22:41:26.066518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.066111    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-kube-api-access-7dl76\" (UniqueName: \"kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-kube-api-access-7dl76\") pod \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") "
Oct 29 22:41:26.066518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.066195    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-credential-socket\") pod \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") "
Oct 29 22:41:26.066518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.066265    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-podinfo\") pod \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") "
Oct 29 22:41:26.066518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.066350    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istiod-ca-cert\") pod \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") "
Oct 29 22:41:26.066518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.066405    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-socket\") pod \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") "
Oct 29 22:41:26.066518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.066476    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-envoy\") pod \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") "
Oct 29 22:41:26.067091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.066586    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-data\") pod \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") "
Oct 29 22:41:26.067091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.066650    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-token\" (UniqueName: \"kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-token\") pod \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\" (UID: \"7a537d76-6b6c-4ae9-a18d-1556b9d26625\") "
Oct 29 22:41:26.070365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.068744    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-socket" pod "7a537d76-6b6c-4ae9-a18d-1556b9d26625" (UID: "7a537d76-6b6c-4ae9-a18d-1556b9d26625"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:26.070365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.069142    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istiod-ca-cert" pod "7a537d76-6b6c-4ae9-a18d-1556b9d26625" (UID: "7a537d76-6b6c-4ae9-a18d-1556b9d26625"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:26.080351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.071941    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-data" pod "7a537d76-6b6c-4ae9-a18d-1556b9d26625" (UID: "7a537d76-6b6c-4ae9-a18d-1556b9d26625"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:26.080351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.072047    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-credential-socket" pod "7a537d76-6b6c-4ae9-a18d-1556b9d26625" (UID: "7a537d76-6b6c-4ae9-a18d-1556b9d26625"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:26.080351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.072140    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-certs" pod "7a537d76-6b6c-4ae9-a18d-1556b9d26625" (UID: "7a537d76-6b6c-4ae9-a18d-1556b9d26625"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:26.105857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.105814    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-kube-api-access-7dl76" pod "7a537d76-6b6c-4ae9-a18d-1556b9d26625" (UID: "7a537d76-6b6c-4ae9-a18d-1556b9d26625"). InnerVolumeSpecName "kube-api-access-7dl76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:26.108797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.108749    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:26.119022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.118985    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-podinfo" pod "7a537d76-6b6c-4ae9-a18d-1556b9d26625" (UID: "7a537d76-6b6c-4ae9-a18d-1556b9d26625"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:26.119159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.119048    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-token" pod "7a537d76-6b6c-4ae9-a18d-1556b9d26625" (UID: "7a537d76-6b6c-4ae9-a18d-1556b9d26625"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:26.120765 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.120739    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-envoy" pod "7a537d76-6b6c-4ae9-a18d-1556b9d26625" (UID: "7a537d76-6b6c-4ae9-a18d-1556b9d26625"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:26.169247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.169214    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:26.169247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.169240    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dl76\" (UniqueName: \"kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-kube-api-access-7dl76\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:26.169247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.169249    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:26.169544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.169258    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:26.169544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.169267    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:26.169544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.169275    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:26.169544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.169283    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:26.169544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.169292    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:26.169544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.169300    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a537d76-6b6c-4ae9-a18d-1556b9d26625-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:26.285888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.285842    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:26.381392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.381301    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"]
Oct 29 22:41:26.382159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.382136    4860 cpu_manager.go:414] "containerMap: removing container" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy"
Oct 29 22:41:26.382159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.382156    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy"
Oct 29 22:41:26.382394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.382307    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy"
Oct 29 22:41:26.382394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.382333    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy"
Oct 29 22:41:26.385179 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.385160    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.405580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.405546    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c7d192-openshift-default-dockercfg-v9rsq\""
Oct 29 22:41:26.409824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.409795    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"]
Oct 29 22:41:26.474450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.474406    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.474596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.474468    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.474596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.474520    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:26.474721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.474607    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.474721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.474640    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.474832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.474802    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.474890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.474844    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn5t2\" (UniqueName: \"kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-kube-api-access-xn5t2\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.474890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.474878    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffef942-b70f-496b-9dd1-aa8921e066ad-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.475121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.475101    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.475217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.475202    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.585345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.579427    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.585345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.579546    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.585345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.579579    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.585345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.579613    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.585345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.579670    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.585345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.579695    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.585345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.579783    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.585345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.579818    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn5t2\" (UniqueName: \"kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-kube-api-access-xn5t2\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.585345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.579850    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffef942-b70f-496b-9dd1-aa8921e066ad-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.585345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.580685    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffef942-b70f-496b-9dd1-aa8921e066ad-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.609348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.587125    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.609348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.587422    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.609348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.587667    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.609348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.587919    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.609348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.593210    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.609348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.593993    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.626375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.624099    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.626375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.625637    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn5t2\" (UniqueName: \"kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-kube-api-access-xn5t2\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.626375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.626284    4860 generic.go:356] "Generic (PLEG): container finished" podID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerID="ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d" exitCode=0
Oct 29 22:41:26.627299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.627277    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"
Oct 29 22:41:26.631095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.628172    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" event={"ID":"7a537d76-6b6c-4ae9-a18d-1556b9d26625","Type":"ContainerDied","Data":"ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d"}
Oct 29 22:41:26.631095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.628235    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7" event={"ID":"7a537d76-6b6c-4ae9-a18d-1556b9d26625","Type":"ContainerDied","Data":"9ec9bbf8f32cd7e1195b713cbcdf4621d4e92ad100e87fb4b8472c61f02a5a9b"}
Oct 29 22:41:26.631095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.628275    4860 scope.go:122] "RemoveContainer" containerID="ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d"
Oct 29 22:41:26.659692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.659663    4860 scope.go:122] "RemoveContainer" containerID="86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493"
Oct 29 22:41:26.671778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.671751    4860 scope.go:122] "RemoveContainer" containerID="ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d"
Oct 29 22:41:26.672087 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:41:26.672061    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d\": container with ID starting with ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d not found: ID does not exist" containerID="ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d"
Oct 29 22:41:26.672131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.672099    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d"} err="failed to get container status \"ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d\": rpc error: code = NotFound desc = could not find container \"ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d\": container with ID starting with ba14739f4f21361b4058b1f90f6d9a3cf535a241abb7bc6abd41cb973c30341d not found: ID does not exist"
Oct 29 22:41:26.672131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.672119    4860 scope.go:122] "RemoveContainer" containerID="86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493"
Oct 29 22:41:26.672640 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:41:26.672375    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493\": container with ID starting with 86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493 not found: ID does not exist" containerID="86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493"
Oct 29 22:41:26.672640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.672399    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493"} err="failed to get container status \"86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493\": rpc error: code = NotFound desc = could not find container \"86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493\": container with ID starting with 86c9763cdae48b0a52d844f6e4f51d4b3d7f000b2b069e9b41b86a986d698493 not found: ID does not exist"
Oct 29 22:41:26.686280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.686254    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:26.700802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.700776    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:26.752480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.752427    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"]
Oct 29 22:41:26.759871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:26.758842    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-39322f-openshift-default-78f75cb6d5-np7f7"]
Oct 29 22:41:27.079924 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:41:27.079895    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffef942_b70f_496b_9dd1_aa8921e066ad.slice/crio-20de642a4d9b690563ecc1d2e2e2286fa39ecdbd4d20a9484e5e73d3cbb00873 WatchSource:0}: Error finding container 20de642a4d9b690563ecc1d2e2e2286fa39ecdbd4d20a9484e5e73d3cbb00873: Status 404 returned error can't find the container with id 20de642a4d9b690563ecc1d2e2e2286fa39ecdbd4d20a9484e5e73d3cbb00873
Oct 29 22:41:27.082939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:27.082910    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"]
Oct 29 22:41:27.083449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:27.083416    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:27.083546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:27.083499    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:27.083613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:27.083547    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:27.466996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:27.466953    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:27.638315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:27.638277    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" event={"ID":"dffef942-b70f-496b-9dd1-aa8921e066ad","Type":"ContainerStarted","Data":"ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac"}
Oct 29 22:41:27.638738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:27.638354    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" event={"ID":"dffef942-b70f-496b-9dd1-aa8921e066ad","Type":"ContainerStarted","Data":"20de642a4d9b690563ecc1d2e2e2286fa39ecdbd4d20a9484e5e73d3cbb00873"}
Oct 29 22:41:27.685259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:27.685210    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" podStartSLOduration=1.6851981710000001 podStartE2EDuration="1.685198171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:27.684730948 +0000 UTC m=+1629.709507102" watchObservedRunningTime="2026-10-29 22:41:27.685198171 +0000 UTC m=+1629.709974302"
Oct 29 22:41:27.701076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:27.701052    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:27.952475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:27.952441    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:28.075391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.075318    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"]
Oct 29 22:41:28.076197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.076176    4860 cpu_manager.go:414] "containerMap: removing container" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy"
Oct 29 22:41:28.076197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.076195    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" containerName="istio-proxy"
Oct 29 22:41:28.079399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.079375    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"
Oct 29 22:41:28.097390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.096807    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:28.109415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.100392    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"]
Oct 29 22:41:28.128563 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.128523    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlpt9\" (UniqueName: \"kubernetes.io/projected/65f1affd-4912-4bdd-b1f9-5d343068e00f-kube-api-access-tlpt9\") pod \"payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq\" (UID: \"65f1affd-4912-4bdd-b1f9-5d343068e00f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"
Oct 29 22:41:28.129617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.129584    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f1affd-4912-4bdd-b1f9-5d343068e00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq\" (UID: \"65f1affd-4912-4bdd-b1f9-5d343068e00f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"
Oct 29 22:41:28.153603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.153572    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a537d76-6b6c-4ae9-a18d-1556b9d26625" path="/var/lib/kubelet/pods/7a537d76-6b6c-4ae9-a18d-1556b9d26625/volumes"
Oct 29 22:41:28.164545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.164514    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a5af18\""
Oct 29 22:41:28.164545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.164524    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a5af18-dockercfg-tztx9\""
Oct 29 22:41:28.194745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.194723    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"]
Oct 29 22:41:28.198272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.198258    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"
Oct 29 22:41:28.218816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.218793    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"]
Oct 29 22:41:28.231408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.231384    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlpt9\" (UniqueName: \"kubernetes.io/projected/65f1affd-4912-4bdd-b1f9-5d343068e00f-kube-api-access-tlpt9\") pod \"payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq\" (UID: \"65f1affd-4912-4bdd-b1f9-5d343068e00f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"
Oct 29 22:41:28.231566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.231547    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz\" (UID: \"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"
Oct 29 22:41:28.231689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.231590    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f1affd-4912-4bdd-b1f9-5d343068e00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq\" (UID: \"65f1affd-4912-4bdd-b1f9-5d343068e00f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"
Oct 29 22:41:28.231784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.231766    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f8sh\" (UniqueName: \"kubernetes.io/projected/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-kube-api-access-4f8sh\") pod \"payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz\" (UID: \"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"
Oct 29 22:41:28.232308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.232280    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f1affd-4912-4bdd-b1f9-5d343068e00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq\" (UID: \"65f1affd-4912-4bdd-b1f9-5d343068e00f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"
Oct 29 22:41:28.245989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.245966    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:28.253241 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.253216    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlpt9\" (UniqueName: \"kubernetes.io/projected/65f1affd-4912-4bdd-b1f9-5d343068e00f-kube-api-access-tlpt9\") pod \"payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq\" (UID: \"65f1affd-4912-4bdd-b1f9-5d343068e00f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"
Oct 29 22:41:28.357352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.353586    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz\" (UID: \"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"
Oct 29 22:41:28.357352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.357108    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f8sh\" (UniqueName: \"kubernetes.io/projected/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-kube-api-access-4f8sh\") pod \"payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz\" (UID: \"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"
Oct 29 22:41:28.368360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.358102    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz\" (UID: \"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"
Oct 29 22:41:28.417370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.395618    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"
Oct 29 22:41:28.417370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.407110    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f8sh\" (UniqueName: \"kubernetes.io/projected/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-kube-api-access-4f8sh\") pod \"payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz\" (UID: \"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"
Oct 29 22:41:28.524392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.516585    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"
Oct 29 22:41:28.710490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.710463    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:28.824519 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:41:28.824491    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f1affd_4912_4bdd_b1f9_5d343068e00f.slice/crio-1f809153850c527b72976e702816de63e312304d44829537500c50628ade6c5c WatchSource:0}: Error finding container 1f809153850c527b72976e702816de63e312304d44829537500c50628ade6c5c: Status 404 returned error can't find the container with id 1f809153850c527b72976e702816de63e312304d44829537500c50628ade6c5c
Oct 29 22:41:28.830663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.830352    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"]
Oct 29 22:41:28.864336 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:41:28.864293    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d11cf5_1e1a_43cb_97b7_72a5c7f7d81d.slice/crio-28a32f4408d94288654b22346fa758137996305af129ab2288872be3b9fff572 WatchSource:0}: Error finding container 28a32f4408d94288654b22346fa758137996305af129ab2288872be3b9fff572: Status 404 returned error can't find the container with id 28a32f4408d94288654b22346fa758137996305af129ab2288872be3b9fff572
Oct 29 22:41:28.867515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:28.867490    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"]
Oct 29 22:41:29.509864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.509831    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"
Oct 29 22:41:29.510031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.509899    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"
Oct 29 22:41:29.662077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.662040    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz" event={"ID":"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d","Type":"ContainerStarted","Data":"14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800"}
Oct 29 22:41:29.662297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.662104    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz" event={"ID":"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d","Type":"ContainerStarted","Data":"28a32f4408d94288654b22346fa758137996305af129ab2288872be3b9fff572"}
Oct 29 22:41:29.662297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.662132    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"
Oct 29 22:41:29.663867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.663843    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq" event={"ID":"65f1affd-4912-4bdd-b1f9-5d343068e00f","Type":"ContainerStarted","Data":"6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f"}
Oct 29 22:41:29.663957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.663886    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq" event={"ID":"65f1affd-4912-4bdd-b1f9-5d343068e00f","Type":"ContainerStarted","Data":"1f809153850c527b72976e702816de63e312304d44829537500c50628ade6c5c"}
Oct 29 22:41:29.663957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.663906    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"
Oct 29 22:41:29.664160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.664143    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:29.665155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.665110    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:41:29.667054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.667022    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:29.778000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.777902    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz" podStartSLOduration=1.777889177 podStartE2EDuration="1.777889177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:29.727712499 +0000 UTC m=+1631.752488653" watchObservedRunningTime="2026-10-29 22:41:29.777889177 +0000 UTC m=+1631.802665326"
Oct 29 22:41:29.782541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:29.782497    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq" podStartSLOduration=1.782483399 podStartE2EDuration="1.782483399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:29.777099486 +0000 UTC m=+1631.801875648" watchObservedRunningTime="2026-10-29 22:41:29.782483399 +0000 UTC m=+1631.807259530"
Oct 29 22:41:30.115844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:30.115721    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:30.252622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:30.252592    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:31.060521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.060489    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:41:31.102653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.102624    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:41:31.201463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.201436    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-certs\") pod \"c5af8478-08a0-4704-b55f-3692764b84e3\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") "
Oct 29 22:41:31.201674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.201472    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-token\" (UniqueName: \"kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-token\") pod \"55380b90-4a75-46b2-ae7b-17f5897ae309\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") "
Oct 29 22:41:31.201674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.201500    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-socket\") pod \"55380b90-4a75-46b2-ae7b-17f5897ae309\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") "
Oct 29 22:41:31.201674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.201561    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55380b90-4a75-46b2-ae7b-17f5897ae309-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55380b90-4a75-46b2-ae7b-17f5897ae309-istiod-ca-cert\") pod \"55380b90-4a75-46b2-ae7b-17f5897ae309\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") "
Oct 29 22:41:31.201876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.201746    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-certs" pod "c5af8478-08a0-4704-b55f-3692764b84e3" (UID: "c5af8478-08a0-4704-b55f-3692764b84e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.201876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.201808    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-certs\") pod \"55380b90-4a75-46b2-ae7b-17f5897ae309\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") "
Oct 29 22:41:31.201995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.201875    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55380b90-4a75-46b2-ae7b-17f5897ae309-istiod-ca-cert" pod "55380b90-4a75-46b2-ae7b-17f5897ae309" (UID: "55380b90-4a75-46b2-ae7b-17f5897ae309"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:31.201995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.201860    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-kube-api-access-tsmzn\" (UniqueName: \"kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-kube-api-access-tsmzn\") pod \"55380b90-4a75-46b2-ae7b-17f5897ae309\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") "
Oct 29 22:41:31.201995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.201955    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-data\") pod \"c5af8478-08a0-4704-b55f-3692764b84e3\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") "
Oct 29 22:41:31.202143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202013    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-envoy\") pod \"c5af8478-08a0-4704-b55f-3692764b84e3\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") "
Oct 29 22:41:31.202143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202034    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-socket" pod "55380b90-4a75-46b2-ae7b-17f5897ae309" (UID: "55380b90-4a75-46b2-ae7b-17f5897ae309"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.202143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202069    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-certs" pod "55380b90-4a75-46b2-ae7b-17f5897ae309" (UID: "55380b90-4a75-46b2-ae7b-17f5897ae309"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.202143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202080    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5af8478-08a0-4704-b55f-3692764b84e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5af8478-08a0-4704-b55f-3692764b84e3-istiod-ca-cert\") pod \"c5af8478-08a0-4704-b55f-3692764b84e3\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") "
Oct 29 22:41:31.202387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202212    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-credential-socket\") pod \"c5af8478-08a0-4704-b55f-3692764b84e3\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") "
Oct 29 22:41:31.202387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202243    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-socket\") pod \"c5af8478-08a0-4704-b55f-3692764b84e3\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") "
Oct 29 22:41:31.202387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202272    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-istio-token\" (UniqueName: \"kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-istio-token\") pod \"c5af8478-08a0-4704-b55f-3692764b84e3\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") "
Oct 29 22:41:31.202387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202355    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-podinfo\") pod \"55380b90-4a75-46b2-ae7b-17f5897ae309\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") "
Oct 29 22:41:31.202611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202391    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5af8478-08a0-4704-b55f-3692764b84e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5af8478-08a0-4704-b55f-3692764b84e3-istio-podinfo\") pod \"c5af8478-08a0-4704-b55f-3692764b84e3\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") "
Oct 29 22:41:31.202611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202448    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-data\") pod \"55380b90-4a75-46b2-ae7b-17f5897ae309\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") "
Oct 29 22:41:31.202611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202526    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-kube-api-access-9twgb\" (UniqueName: \"kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-kube-api-access-9twgb\") pod \"c5af8478-08a0-4704-b55f-3692764b84e3\" (UID: \"c5af8478-08a0-4704-b55f-3692764b84e3\") "
Oct 29 22:41:31.202611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202565    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-credential-socket\") pod \"55380b90-4a75-46b2-ae7b-17f5897ae309\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") "
Oct 29 22:41:31.202825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202723    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-envoy\") pod \"55380b90-4a75-46b2-ae7b-17f5897ae309\" (UID: \"55380b90-4a75-46b2-ae7b-17f5897ae309\") "
Oct 29 22:41:31.203283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202443    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-credential-socket" pod "c5af8478-08a0-4704-b55f-3692764b84e3" (UID: "c5af8478-08a0-4704-b55f-3692764b84e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.203445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202547    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5af8478-08a0-4704-b55f-3692764b84e3-istiod-ca-cert" pod "c5af8478-08a0-4704-b55f-3692764b84e3" (UID: "c5af8478-08a0-4704-b55f-3692764b84e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:31.203445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.202654    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-data" pod "c5af8478-08a0-4704-b55f-3692764b84e3" (UID: "c5af8478-08a0-4704-b55f-3692764b84e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.203445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.203108    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-socket" pod "c5af8478-08a0-4704-b55f-3692764b84e3" (UID: "c5af8478-08a0-4704-b55f-3692764b84e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.204216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.204188    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-credential-socket" pod "55380b90-4a75-46b2-ae7b-17f5897ae309" (UID: "55380b90-4a75-46b2-ae7b-17f5897ae309"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.204815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.204764    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-envoy" pod "c5af8478-08a0-4704-b55f-3692764b84e3" (UID: "c5af8478-08a0-4704-b55f-3692764b84e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.204911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.204846    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-kube-api-access-tsmzn" pod "55380b90-4a75-46b2-ae7b-17f5897ae309" (UID: "55380b90-4a75-46b2-ae7b-17f5897ae309"). InnerVolumeSpecName "kube-api-access-tsmzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.205284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205261    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-data" pod "55380b90-4a75-46b2-ae7b-17f5897ae309" (UID: "55380b90-4a75-46b2-ae7b-17f5897ae309"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.205418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205392    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-token" pod "55380b90-4a75-46b2-ae7b-17f5897ae309" (UID: "55380b90-4a75-46b2-ae7b-17f5897ae309"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.205677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205637    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.205677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205662    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.205677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205675    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.205862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205688    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.205862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205700    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.205862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205714    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.205862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205727    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.205862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205739    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55380b90-4a75-46b2-ae7b-17f5897ae309-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.205862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205752    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.205862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205765    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsmzn\" (UniqueName: \"kubernetes.io/projected/55380b90-4a75-46b2-ae7b-17f5897ae309-kube-api-access-tsmzn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.205862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205772    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5af8478-08a0-4704-b55f-3692764b84e3-istio-podinfo" pod "c5af8478-08a0-4704-b55f-3692764b84e3" (UID: "c5af8478-08a0-4704-b55f-3692764b84e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:31.205862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205778    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.205862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205794    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5af8478-08a0-4704-b55f-3692764b84e3-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.205862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.205806    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5af8478-08a0-4704-b55f-3692764b84e3-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.206560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.206406    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-kube-api-access-9twgb" pod "c5af8478-08a0-4704-b55f-3692764b84e3" (UID: "c5af8478-08a0-4704-b55f-3692764b84e3"). InnerVolumeSpecName "kube-api-access-9twgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.206739 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.206712    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-envoy" pod "55380b90-4a75-46b2-ae7b-17f5897ae309" (UID: "55380b90-4a75-46b2-ae7b-17f5897ae309"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:31.207065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.207040    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-istio-token" pod "c5af8478-08a0-4704-b55f-3692764b84e3" (UID: "c5af8478-08a0-4704-b55f-3692764b84e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:31.207488 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.207468    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-podinfo" pod "55380b90-4a75-46b2-ae7b-17f5897ae309" (UID: "55380b90-4a75-46b2-ae7b-17f5897ae309"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:31.307838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.307762    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.307838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.307793    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.307838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.307802    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5af8478-08a0-4704-b55f-3692764b84e3-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.307838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.307811    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9twgb\" (UniqueName: \"kubernetes.io/projected/c5af8478-08a0-4704-b55f-3692764b84e3-kube-api-access-9twgb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.307838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.307820    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55380b90-4a75-46b2-ae7b-17f5897ae309-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:31.662235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.662078    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:31.685393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.685366    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"
Oct 29 22:41:31.685393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.685395    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9" event={"ID":"55380b90-4a75-46b2-ae7b-17f5897ae309","Type":"ContainerDied","Data":"09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807"}
Oct 29 22:41:31.685663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.685286    4860 generic.go:356] "Generic (PLEG): container finished" podID="55380b90-4a75-46b2-ae7b-17f5897ae309" containerID="09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807" exitCode=0
Oct 29 22:41:31.685663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.685466    4860 scope.go:122] "RemoveContainer" containerID="09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807"
Oct 29 22:41:31.685663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.685567    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9" event={"ID":"55380b90-4a75-46b2-ae7b-17f5897ae309","Type":"ContainerDied","Data":"d13d4f452f316a65185dd7d214e53870fbec9c4bd10e3044ce92b1f053d5a488"}
Oct 29 22:41:31.687963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.687935    4860 generic.go:356] "Generic (PLEG): container finished" podID="c5af8478-08a0-4704-b55f-3692764b84e3" containerID="720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35" exitCode=0
Oct 29 22:41:31.688094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.687992    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"
Oct 29 22:41:31.688094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.688042    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" event={"ID":"c5af8478-08a0-4704-b55f-3692764b84e3","Type":"ContainerDied","Data":"720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35"}
Oct 29 22:41:31.688233 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.688116    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d" event={"ID":"c5af8478-08a0-4704-b55f-3692764b84e3","Type":"ContainerDied","Data":"a228fa455a3baa8ed2f187f024ee2b071e864dc7f426adfece0ebcdd8387446b"}
Oct 29 22:41:31.709449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.709267    4860 scope.go:122] "RemoveContainer" containerID="09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807"
Oct 29 22:41:31.709753 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:41:31.709730    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807\": container with ID starting with 09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807 not found: ID does not exist" containerID="09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807"
Oct 29 22:41:31.709857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.709764    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807"} err="failed to get container status \"09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807\": rpc error: code = NotFound desc = could not find container \"09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807\": container with ID starting with 09dcad98a1732d99abe49da36d26815dace2069353dbebcaf21e8222b5c73807 not found: ID does not exist"
Oct 29 22:41:31.709857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.709786    4860 scope.go:122] "RemoveContainer" containerID="720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35"
Oct 29 22:41:31.720119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.720100    4860 scope.go:122] "RemoveContainer" containerID="5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2"
Oct 29 22:41:31.730489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.730473    4860 scope.go:122] "RemoveContainer" containerID="720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35"
Oct 29 22:41:31.730755 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:41:31.730737    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35\": container with ID starting with 720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35 not found: ID does not exist" containerID="720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35"
Oct 29 22:41:31.730813 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.730760    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35"} err="failed to get container status \"720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35\": rpc error: code = NotFound desc = could not find container \"720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35\": container with ID starting with 720ff347283af05822441d723f3b8c77abcb8cdf35555a4177bf4af2d55e7c35 not found: ID does not exist"
Oct 29 22:41:31.730813 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.730786    4860 scope.go:122] "RemoveContainer" containerID="5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2"
Oct 29 22:41:31.731036 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:41:31.731018    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2\": container with ID starting with 5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2 not found: ID does not exist" containerID="5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2"
Oct 29 22:41:31.731088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.731042    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2"} err="failed to get container status \"5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2\": rpc error: code = NotFound desc = could not find container \"5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2\": container with ID starting with 5da80f43277a23349895824ade3363afbd04c3d1cce0cfffd0ddcd77de8b77b2 not found: ID does not exist"
Oct 29 22:41:31.777403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.777372    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"]
Oct 29 22:41:31.790069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.790038    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-mjcl9"]
Oct 29 22:41:31.835011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.834973    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"]
Oct 29 22:41:31.854387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.854364    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1fa4b8b6-openshift-default-6765496769-47v7d"]
Oct 29 22:41:31.880952 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:31.880924    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:32.032600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.032570    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:32.198344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.187576    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55380b90-4a75-46b2-ae7b-17f5897ae309" path="/var/lib/kubelet/pods/55380b90-4a75-46b2-ae7b-17f5897ae309/volumes"
Oct 29 22:41:32.198344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.188358    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" path="/var/lib/kubelet/pods/c5af8478-08a0-4704-b55f-3692764b84e3/volumes"
Oct 29 22:41:32.371785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.371711    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"]
Oct 29 22:41:32.372523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.372506    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy"
Oct 29 22:41:32.372574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.372524    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy"
Oct 29 22:41:32.372574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.372544    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy"
Oct 29 22:41:32.372574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.372553    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy"
Oct 29 22:41:32.372574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.372561    4860 cpu_manager.go:414] "containerMap: removing container" podUID="55380b90-4a75-46b2-ae7b-17f5897ae309" containerName="istio-proxy"
Oct 29 22:41:32.372574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.372566    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55380b90-4a75-46b2-ae7b-17f5897ae309" containerName="istio-proxy"
Oct 29 22:41:32.372731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.372693    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy"
Oct 29 22:41:32.372731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.372703    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="55380b90-4a75-46b2-ae7b-17f5897ae309" containerName="istio-proxy"
Oct 29 22:41:32.377065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.377020    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:32.399876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.399847    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"]
Oct 29 22:41:32.401859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.401834    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:41:32.402008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.401901    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:41:32.577831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.577801    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-065d84b9-47cwd\" (UID: \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:32.578036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.578017    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvgs\" (UniqueName: \"kubernetes.io/projected/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-kube-api-access-tdvgs\") pod \"maas-api-revoke-sub-065d84b9-47cwd\" (UID: \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:32.578125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.578048    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-xks-ca\") pod \"maas-api-revoke-sub-065d84b9-47cwd\" (UID: \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:32.680084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.679999    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-065d84b9-47cwd\" (UID: \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:32.680256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.680219    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvgs\" (UniqueName: \"kubernetes.io/projected/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-kube-api-access-tdvgs\") pod \"maas-api-revoke-sub-065d84b9-47cwd\" (UID: \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:32.680359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.680250    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-xks-ca\") pod \"maas-api-revoke-sub-065d84b9-47cwd\" (UID: \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:32.680657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.680637    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-065d84b9-47cwd\" (UID: \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:32.682654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.682630    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-xks-ca\") pod \"maas-api-revoke-sub-065d84b9-47cwd\" (UID: \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:32.701737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.701715    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvgs\" (UniqueName: \"kubernetes.io/projected/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-kube-api-access-tdvgs\") pod \"maas-api-revoke-sub-065d84b9-47cwd\" (UID: \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:32.996973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:32.996896    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:33.182389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:33.182354    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:33.424591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:33.424492    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"]
Oct 29 22:41:33.444184 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:41:33.444149    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb59bda_b92f_4c8c_8819_9c5a2ddc3ded.slice/crio-282eb19d4864f0c3bef11e17583c8dd0b1f1733905e362b7d7ce19181f9e18b9 WatchSource:0}: Error finding container 282eb19d4864f0c3bef11e17583c8dd0b1f1733905e362b7d7ce19181f9e18b9: Status 404 returned error can't find the container with id 282eb19d4864f0c3bef11e17583c8dd0b1f1733905e362b7d7ce19181f9e18b9
Oct 29 22:41:33.711309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:33.711268    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd" event={"ID":"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded","Type":"ContainerStarted","Data":"282eb19d4864f0c3bef11e17583c8dd0b1f1733905e362b7d7ce19181f9e18b9"}
Oct 29 22:41:33.748874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:33.748842    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:34.723091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:34.723063    4860 generic.go:356] "Generic (PLEG): container finished" podID="deb59bda-b92f-4c8c-8819-9c5a2ddc3ded" containerID="cd890ffce2c83796dfc8a0bd27cefe0fac30c655498e99c5c8a3bc62e8601703" exitCode=0
Oct 29 22:41:34.723527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:34.723145    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd" event={"ID":"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded","Type":"ContainerDied","Data":"cd890ffce2c83796dfc8a0bd27cefe0fac30c655498e99c5c8a3bc62e8601703"}
Oct 29 22:41:35.890915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:35.890874    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:35.934356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:35.934316    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:36.034908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:36.034834    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-xks-ca\") pod \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\" (UID: \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\") "
Oct 29 22:41:36.034908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:36.034886    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-openshift-ca\") pod \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\" (UID: \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\") "
Oct 29 22:41:36.034908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:36.034905    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-kube-api-access-tdvgs\" (UniqueName: \"kubernetes.io/projected/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-kube-api-access-tdvgs\") pod \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\" (UID: \"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded\") "
Oct 29 22:41:36.035270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:36.035247    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-openshift-ca" pod "deb59bda-b92f-4c8c-8819-9c5a2ddc3ded" (UID: "deb59bda-b92f-4c8c-8819-9c5a2ddc3ded"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:36.035636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:36.035618    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:36.036854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:36.036830    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-xks-ca" pod "deb59bda-b92f-4c8c-8819-9c5a2ddc3ded" (UID: "deb59bda-b92f-4c8c-8819-9c5a2ddc3ded"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:41:36.037075 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:36.037053    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-kube-api-access-tdvgs" pod "deb59bda-b92f-4c8c-8819-9c5a2ddc3ded" (UID: "deb59bda-b92f-4c8c-8819-9c5a2ddc3ded"). InnerVolumeSpecName "kube-api-access-tdvgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:36.137190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:36.137169    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:36.137190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:36.137189    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdvgs\" (UniqueName: \"kubernetes.io/projected/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded-kube-api-access-tdvgs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:36.742196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:36.742173    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"
Oct 29 22:41:36.742379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:36.742169    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd" event={"ID":"deb59bda-b92f-4c8c-8819-9c5a2ddc3ded","Type":"ContainerDied","Data":"282eb19d4864f0c3bef11e17583c8dd0b1f1733905e362b7d7ce19181f9e18b9"}
Oct 29 22:41:36.742379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:36.742288    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="282eb19d4864f0c3bef11e17583c8dd0b1f1733905e362b7d7ce19181f9e18b9"
Oct 29 22:41:37.397460 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:37.397429    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:37.891312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:37.891280    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"]
Oct 29 22:41:37.891523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:37.891492    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" podUID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" containerName="istio-proxy" containerID="cri-o://6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125" gracePeriod=30
Oct 29 22:41:39.013198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:39.013158    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:39.265950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:39.265868    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:40.003564 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:40.003524    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:40.115549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:40.115519    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:40.676810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:40.676784    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"
Oct 29 22:41:40.676997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:40.676847    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"
Oct 29 22:41:40.903995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:40.903955    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:42.427969 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:42.427937    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:43.085277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.085206    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:43.169312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.169256    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:41:43.253181 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.253148    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-certs\") pod \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") "
Oct 29 22:41:43.253400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.253381    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-podinfo\") pod \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") "
Oct 29 22:41:43.253504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.253450    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-socket\") pod \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") "
Oct 29 22:41:43.253504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.253488    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-kube-api-access-dlvkd\" (UniqueName: \"kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-kube-api-access-dlvkd\") pod \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") "
Oct 29 22:41:43.253624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.253518    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-data\") pod \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") "
Oct 29 22:41:43.253624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.253547    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-certs" pod "4c2aad81-e61d-4edb-85c8-0c8e725d00a5" (UID: "4c2aad81-e61d-4edb-85c8-0c8e725d00a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:43.253624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.253560    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-envoy\") pod \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") "
Oct 29 22:41:43.253784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.253614    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-token\" (UniqueName: \"kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-token\") pod \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") "
Oct 29 22:41:43.253784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.253718    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istiod-ca-cert\") pod \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") "
Oct 29 22:41:43.253889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.253820    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-credential-socket\") pod \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\" (UID: \"4c2aad81-e61d-4edb-85c8-0c8e725d00a5\") "
Oct 29 22:41:43.254696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.254669    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:43.254977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.254951    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-credential-socket" pod "4c2aad81-e61d-4edb-85c8-0c8e725d00a5" (UID: "4c2aad81-e61d-4edb-85c8-0c8e725d00a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:43.255082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.255052    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-socket" pod "4c2aad81-e61d-4edb-85c8-0c8e725d00a5" (UID: "4c2aad81-e61d-4edb-85c8-0c8e725d00a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:43.255145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.254876    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-data" pod "4c2aad81-e61d-4edb-85c8-0c8e725d00a5" (UID: "4c2aad81-e61d-4edb-85c8-0c8e725d00a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:43.255300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.255274    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istiod-ca-cert" pod "4c2aad81-e61d-4edb-85c8-0c8e725d00a5" (UID: "4c2aad81-e61d-4edb-85c8-0c8e725d00a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:41:43.256310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.256263    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-podinfo" pod "4c2aad81-e61d-4edb-85c8-0c8e725d00a5" (UID: "4c2aad81-e61d-4edb-85c8-0c8e725d00a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:41:43.258211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.258189    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-token" pod "4c2aad81-e61d-4edb-85c8-0c8e725d00a5" (UID: "4c2aad81-e61d-4edb-85c8-0c8e725d00a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:43.258392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.258270    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-kube-api-access-dlvkd" pod "4c2aad81-e61d-4edb-85c8-0c8e725d00a5" (UID: "4c2aad81-e61d-4edb-85c8-0c8e725d00a5"). InnerVolumeSpecName "kube-api-access-dlvkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:41:43.258587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.258560    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-envoy" pod "4c2aad81-e61d-4edb-85c8-0c8e725d00a5" (UID: "4c2aad81-e61d-4edb-85c8-0c8e725d00a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:41:43.356363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.356316    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:43.356363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.356363    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:43.356553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.356377    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlvkd\" (UniqueName: \"kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-kube-api-access-dlvkd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:43.356553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.356395    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:43.356553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.356411    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:43.356553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.356420    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:43.356553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.356429    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:43.356553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.356437    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c2aad81-e61d-4edb-85c8-0c8e725d00a5-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:41:43.469531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.469499    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"]
Oct 29 22:41:43.470664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.470642    4860 cpu_manager.go:414] "containerMap: removing container" podUID="deb59bda-b92f-4c8c-8819-9c5a2ddc3ded" containerName="revoke-keys"
Oct 29 22:41:43.470664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.470663    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb59bda-b92f-4c8c-8819-9c5a2ddc3ded" containerName="revoke-keys"
Oct 29 22:41:43.470860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.470713    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" containerName="istio-proxy"
Oct 29 22:41:43.470860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.470718    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" containerName="istio-proxy"
Oct 29 22:41:43.470860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.470847    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb59bda-b92f-4c8c-8819-9c5a2ddc3ded" containerName="revoke-keys"
Oct 29 22:41:43.471043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.470863    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5af8478-08a0-4704-b55f-3692764b84e3" containerName="istio-proxy"
Oct 29 22:41:43.471043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.470872    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" containerName="istio-proxy"
Oct 29 22:41:43.473926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.473906    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.494861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.494833    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"]
Oct 29 22:41:43.500555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.500534    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-9aafe6-openshift-default-dockercfg-6g2nq\""
Oct 29 22:41:43.539174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.539146    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:43.559240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.559208    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-istio-token\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.559418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.559357    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8790a27-ef61-43db-803b-fc295959b33a-istio-podinfo\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.559418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.559398    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8790a27-ef61-43db-803b-fc295959b33a-istiod-ca-cert\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.559565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.559483    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6vlm\" (UniqueName: \"kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-kube-api-access-q6vlm\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.559611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.559587    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-envoy\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.559657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.559638    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.559712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.559677    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-certs\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.559800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.559787    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-credential-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.559867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.559856    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-data\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.667959    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-certs\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.668034    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-credential-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.668130    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-data\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.668279    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-istio-token\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.668384    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8790a27-ef61-43db-803b-fc295959b33a-istio-podinfo\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.668440    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8790a27-ef61-43db-803b-fc295959b33a-istiod-ca-cert\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.668470    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6vlm\" (UniqueName: \"kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-kube-api-access-q6vlm\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.668534    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-envoy\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.668584    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.669098    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.669371    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-certs\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.669657    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-credential-socket\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.670344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.669916    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-data\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.671252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.671228    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8790a27-ef61-43db-803b-fc295959b33a-istiod-ca-cert\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.685728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.685688    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-envoy\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.702337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.702166    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:43.715772 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.715687    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-istio-token\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.723082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.723006    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8790a27-ef61-43db-803b-fc295959b33a-istio-podinfo\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.726594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.725827    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6vlm\" (UniqueName: \"kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-kube-api-access-q6vlm\") pod \"e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") " pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.799349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.796400    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:43.833470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.833425    4860 generic.go:356] "Generic (PLEG): container finished" podID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" containerID="6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125" exitCode=0
Oct 29 22:41:43.833937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.833910    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" event={"ID":"4c2aad81-e61d-4edb-85c8-0c8e725d00a5","Type":"ContainerDied","Data":"6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125"}
Oct 29 22:41:43.834189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.834170    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd" event={"ID":"4c2aad81-e61d-4edb-85c8-0c8e725d00a5","Type":"ContainerDied","Data":"2a2cb279d631899420c12df4b9a3c43d268df45a49be2f53c101b15cf692edc1"}
Oct 29 22:41:43.834351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.834317    4860 scope.go:122] "RemoveContainer" containerID="6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125"
Oct 29 22:41:43.834704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.834687    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"
Oct 29 22:41:43.853138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.853120    4860 scope.go:122] "RemoveContainer" containerID="6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125"
Oct 29 22:41:43.853611 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:41:43.853589    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125\": container with ID starting with 6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125 not found: ID does not exist" containerID="6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125"
Oct 29 22:41:43.853682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.853619    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125"} err="failed to get container status \"6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125\": rpc error: code = NotFound desc = could not find container \"6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125\": container with ID starting with 6bceabf5226186f34d5e597079091dd5813277228f3afef8760fcbf5749dc125 not found: ID does not exist"
Oct 29 22:41:43.899890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.899858    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"]
Oct 29 22:41:43.905448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:43.905421    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-p6swd"]
Oct 29 22:41:44.114846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:44.114775    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2aad81-e61d-4edb-85c8-0c8e725d00a5" path="/var/lib/kubelet/pods/4c2aad81-e61d-4edb-85c8-0c8e725d00a5/volumes"
Oct 29 22:41:44.140962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:44.140930    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"]
Oct 29 22:41:44.143674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:44.143585    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:44.144817 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:44.144781    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:44.144953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:44.144858    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:44.144953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:44.144902    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:44.336568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:44.336503    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:44.478844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:44.478800    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:44.844293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:44.844213    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" event={"ID":"c8790a27-ef61-43db-803b-fc295959b33a","Type":"ContainerStarted","Data":"8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72"}
Oct 29 22:41:44.844293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:44.844280    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" event={"ID":"c8790a27-ef61-43db-803b-fc295959b33a","Type":"ContainerStarted","Data":"18b8fb915f32b6a0d91e6ce22f42c42f4d34ee63855ea50ffa457690f420d1ed"}
Oct 29 22:41:44.918210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:44.918154    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" podStartSLOduration=1.918141909 podStartE2EDuration="1.918141909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:44.90828399 +0000 UTC m=+1646.933060141" watchObservedRunningTime="2026-10-29 22:41:44.918141909 +0000 UTC m=+1646.942918040"
Oct 29 22:41:45.797402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:45.797369    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:45.803734 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:45.803708    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:45.854399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:45.854360    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:45.855150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:45.855131    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:41:45.857160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:45.857136    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:45.965885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:45.965851    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"]
Oct 29 22:41:45.969614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:45.969594    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"
Oct 29 22:41:45.986196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:45.986167    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:45.986354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:45.986338    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5a69c9\""
Oct 29 22:41:45.986460 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:45.986440    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5a69c9-dockercfg-dlddn\""
Oct 29 22:41:45.987064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:45.987042    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"]
Oct 29 22:41:46.006896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.006864    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smmgt\" (UniqueName: \"kubernetes.io/projected/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-kube-api-access-smmgt\") pod \"payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg\" (UID: \"4bd45d1b-91fb-44fb-95c4-17e38f9d905f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"
Oct 29 22:41:46.007115 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.007088    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg\" (UID: \"4bd45d1b-91fb-44fb-95c4-17e38f9d905f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"
Oct 29 22:41:46.116395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.113001    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smmgt\" (UniqueName: \"kubernetes.io/projected/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-kube-api-access-smmgt\") pod \"payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg\" (UID: \"4bd45d1b-91fb-44fb-95c4-17e38f9d905f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"
Oct 29 22:41:46.116395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.113171    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg\" (UID: \"4bd45d1b-91fb-44fb-95c4-17e38f9d905f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"
Oct 29 22:41:46.116395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.114181    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg\" (UID: \"4bd45d1b-91fb-44fb-95c4-17e38f9d905f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"
Oct 29 22:41:46.226346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.221163    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smmgt\" (UniqueName: \"kubernetes.io/projected/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-kube-api-access-smmgt\") pod \"payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg\" (UID: \"4bd45d1b-91fb-44fb-95c4-17e38f9d905f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"
Oct 29 22:41:46.286484 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.286454    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"
Oct 29 22:41:46.455740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.455706    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:46.582257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.582225    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:46.605543 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:41:46.605513    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd45d1b_91fb_44fb_95c4_17e38f9d905f.slice/crio-9253377fa5f7c788af2094252952ce5a0c6a3d3d2c7b1d368816904b6b559433 WatchSource:0}: Error finding container 9253377fa5f7c788af2094252952ce5a0c6a3d3d2c7b1d368816904b6b559433: Status 404 returned error can't find the container with id 9253377fa5f7c788af2094252952ce5a0c6a3d3d2c7b1d368816904b6b559433
Oct 29 22:41:46.607446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.607423    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"]
Oct 29 22:41:46.816574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.816506    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:46.865202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.865172    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg" event={"ID":"4bd45d1b-91fb-44fb-95c4-17e38f9d905f","Type":"ContainerStarted","Data":"0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880"}
Oct 29 22:41:46.865352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.865224    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg" event={"ID":"4bd45d1b-91fb-44fb-95c4-17e38f9d905f","Type":"ContainerStarted","Data":"9253377fa5f7c788af2094252952ce5a0c6a3d3d2c7b1d368816904b6b559433"}
Oct 29 22:41:46.865510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.865492    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"
Oct 29 22:41:46.905790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:46.905750    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg" podStartSLOduration=1.905737438 podStartE2EDuration="1.905737438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:46.905444507 +0000 UTC m=+1648.930220639" watchObservedRunningTime="2026-10-29 22:41:46.905737438 +0000 UTC m=+1648.930513568"
Oct 29 22:41:47.655312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:47.655279    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:47.829350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:47.829297    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:47.945537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:47.945498    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"]
Oct 29 22:41:47.949688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:47.949665    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:47.975585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:47.975555    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"]
Oct 29 22:41:47.982594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:47.982569    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4d3168-openshift-default-dockercfg-88hm9\""
Oct 29 22:41:48.056935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.056878    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-envoy\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.057187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.057168    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-certs\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.057313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.057302    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-token\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.057493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.057474    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-podinfo\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.057614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.057598    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.058111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.058090    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-credential-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.058384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.058369    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e292294d-64b9-43b7-a0b9-89f63da16cdb-istiod-ca-cert\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.058889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.058866    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-data\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.059026 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.059012    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8f8h\" (UniqueName: \"kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-kube-api-access-b8f8h\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.066474 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:41:48.066450    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:41:48.160979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.160947    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-envoy\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.160979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.160984    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-certs\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.161220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.161067    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-token\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.161220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.161156    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-podinfo\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.161220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.161215    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.161414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.161269    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-certs\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.161414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.161283    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-credential-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.161489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.161431    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e292294d-64b9-43b7-a0b9-89f63da16cdb-istiod-ca-cert\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.161533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.161483    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-data\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.161578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.161530    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8f8h\" (UniqueName: \"kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-kube-api-access-b8f8h\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.161578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.161550    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.161704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.161591    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-credential-socket\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.161938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.161912    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-data\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.162063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.162046    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e292294d-64b9-43b7-a0b9-89f63da16cdb-istiod-ca-cert\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.163298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.163277    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-envoy\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.163664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.163647    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-podinfo\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.183849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.183822    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-token\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.184867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.184848    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8f8h\" (UniqueName: \"kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-kube-api-access-b8f8h\") pod \"e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") " pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.268166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.268041    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:48.447862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.447833    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:48.598026 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:41:48.597995    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode292294d_64b9_43b7_a0b9_89f63da16cdb.slice/crio-9f609eecc763d08d60e0d6580b0f6248d2916aca4f697aaaeea86255e195b0ba WatchSource:0}: Error finding container 9f609eecc763d08d60e0d6580b0f6248d2916aca4f697aaaeea86255e195b0ba: Status 404 returned error can't find the container with id 9f609eecc763d08d60e0d6580b0f6248d2916aca4f697aaaeea86255e195b0ba
Oct 29 22:41:48.600939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.600899    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:48.601015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.600976    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:48.601015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.601005    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:41:48.601581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.601473    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"]
Oct 29 22:41:48.901862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.901769    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" event={"ID":"e292294d-64b9-43b7-a0b9-89f63da16cdb","Type":"ContainerStarted","Data":"b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64"}
Oct 29 22:41:48.901862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.901830    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" event={"ID":"e292294d-64b9-43b7-a0b9-89f63da16cdb","Type":"ContainerStarted","Data":"9f609eecc763d08d60e0d6580b0f6248d2916aca4f697aaaeea86255e195b0ba"}
Oct 29 22:41:48.965696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:48.965642    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" podStartSLOduration=1.9656289 podStartE2EDuration="1.9656289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:48.964943679 +0000 UTC m=+1650.989719821" watchObservedRunningTime="2026-10-29 22:41:48.9656289 +0000 UTC m=+1650.990405033"
Oct 29 22:41:49.269059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:49.269029    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:49.270947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:49.270915    4860 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 29 22:41:49.271076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:49.270967    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" podUID="e292294d-64b9-43b7-a0b9-89f63da16cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 29 22:41:49.766812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:49.766783    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:49.989348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:49.977599    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:50.314452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:50.312464    4860 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 29 22:41:50.314452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:50.312591    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" podUID="e292294d-64b9-43b7-a0b9-89f63da16cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 29 22:41:50.703369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:50.703078    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:51.292480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:51.292436    4860 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 29 22:41:51.292885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:51.292499    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" podUID="e292294d-64b9-43b7-a0b9-89f63da16cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 29 22:41:52.280462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:52.279826    4860 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 29 22:41:52.280462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:52.279885    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" podUID="e292294d-64b9-43b7-a0b9-89f63da16cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 29 22:41:52.335455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:52.335415    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:53.272670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:53.272633    4860 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 29 22:41:53.272953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:53.272921    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" podUID="e292294d-64b9-43b7-a0b9-89f63da16cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 29 22:41:54.038978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:54.038946    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"]
Oct 29 22:41:54.052388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:54.052360    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99ca7ac1-xvdb9"]
Oct 29 22:41:54.126360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:54.124750    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:54.144539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:54.144505    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de0281ae-c434-4ebd-b8b3-bb6cf61693c3" path="/var/lib/kubelet/pods/de0281ae-c434-4ebd-b8b3-bb6cf61693c3/volumes"
Oct 29 22:41:54.278686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:54.278654    4860 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 29 22:41:54.278979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:54.278947    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" podUID="e292294d-64b9-43b7-a0b9-89f63da16cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 29 22:41:55.268337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:55.268293    4860 patch_prober.go:29] interesting pod/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Oct 29 22:41:55.268761 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:55.268358    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" podUID="e292294d-64b9-43b7-a0b9-89f63da16cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Oct 29 22:41:55.958978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:55.958943    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:56.274074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:56.273993    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:56.274561 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:56.274356    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:56.275525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:56.275503    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:41:57.567451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:57.567420    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:57.695168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:57.695143    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:57.819602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:57.819522    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:57.888602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:57.888576    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"
Oct 29 22:41:59.313603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:41:59.313573    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:01.049566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:01.049531    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:02.782379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:02.782317    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:02.982010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:02.981971    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:03.144730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:03.144623    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:04.463432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:04.463392    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:06.254254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:06.254225    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:07.851877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:07.851844    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:09.395090 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:09.395045    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:11.027387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:11.027306    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:12.385108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:12.385070    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:12.689916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:12.689886    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:14.441785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:14.441756    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:15.974449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:15.974413    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:17.587768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:17.587734    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:17.708426 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:17.708391    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:17.838100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:17.838019    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:18.056988 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:42:18.056950    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:42:19.215823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.215786    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"]
Oct 29 22:42:19.221839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.221818    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.241596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.241567    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"]
Oct 29 22:42:19.249836 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.249796    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c296a3-openshift-default-dockercfg-p5jk9\""
Oct 29 22:42:19.316128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.316097    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-certs\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.316313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.316298    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-envoy\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.316389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.316361    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-socket\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.316430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.316395    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-data\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.316430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.316420    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-token\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.316498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.316487    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46eab5e9-0a8b-4533-a604-d3478d41989b-istiod-ca-cert\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.316544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.316532    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97c64\" (UniqueName: \"kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-kube-api-access-97c64\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.316581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.316553    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-credential-socket\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.316639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.316620    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-podinfo\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.417736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.417705    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-envoy\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.417884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.417755    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-socket\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.417884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.417777    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-data\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.417884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.417800    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-token\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.417884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.417828    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46eab5e9-0a8b-4533-a604-d3478d41989b-istiod-ca-cert\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.418108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.417883    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97c64\" (UniqueName: \"kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-kube-api-access-97c64\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.418108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.417907    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-credential-socket\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.418108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.418010    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-podinfo\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.418273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.418171    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-certs\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.418273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.418249    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-socket\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.418273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.418257    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-data\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.418501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.418268    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-credential-socket\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.418613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.418581    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-certs\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.418876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.418858    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46eab5e9-0a8b-4533-a604-d3478d41989b-istiod-ca-cert\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.420218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.420194    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-envoy\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.420652 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.420633    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-podinfo\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.447574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.447548    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-token\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.447690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.447582    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97c64\" (UniqueName: \"kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-kube-api-access-97c64\") pod \"e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") " pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.536169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.536084    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:19.539800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.539774    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:19.871834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.871794    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:19.871966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.871865    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:19.871966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.871899    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:19.873018 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.872992    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"]
Oct 29 22:42:19.923474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.923447    4860 scope.go:122] "RemoveContainer" containerID="c08b0189edc4c326a2665959a08f241f3808de327c8451db562ad2e07c4d219f"
Oct 29 22:42:19.942217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:19.942196    4860 scope.go:122] "RemoveContainer" containerID="403af61a7f99e774d815ec752efba0a9265f12701194526a652e19195a7c63fe"
Oct 29 22:42:20.231653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:20.231608    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9" event={"ID":"46eab5e9-0a8b-4533-a604-d3478d41989b","Type":"ContainerStarted","Data":"a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9"}
Oct 29 22:42:20.232045 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:20.231669    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9" event={"ID":"46eab5e9-0a8b-4533-a604-d3478d41989b","Type":"ContainerStarted","Data":"4cbb2ad1b0224dabdb442e350bac748878d04d356bc25c389e3021cc9c04d10c"}
Oct 29 22:42:20.304415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:20.304368    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9" podStartSLOduration=1.304354148 podStartE2EDuration="1.304354148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:20.290134514 +0000 UTC m=+1682.314910679" watchObservedRunningTime="2026-10-29 22:42:20.304354148 +0000 UTC m=+1682.329130279"
Oct 29 22:42:20.537249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:20.537166    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:20.539346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:20.539294    4860 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 29 22:42:20.539477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:20.539373    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9" podUID="46eab5e9-0a8b-4533-a604-d3478d41989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 29 22:42:21.203601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:21.203553    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:21.345647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:21.345588    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:21.537490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:21.537406    4860 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 29 22:42:21.537490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:21.537463    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9" podUID="46eab5e9-0a8b-4533-a604-d3478d41989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 29 22:42:22.536786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:22.536750    4860 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 29 22:42:22.537295 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:22.536808    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9" podUID="46eab5e9-0a8b-4533-a604-d3478d41989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 29 22:42:22.834729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:22.834646    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:23.446915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:23.446884    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:23.536893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:23.536864    4860 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 29 22:42:23.537305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:23.536913    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9" podUID="46eab5e9-0a8b-4533-a604-d3478d41989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 29 22:42:24.536909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:24.536870    4860 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 29 22:42:24.537313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:24.536953    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9" podUID="46eab5e9-0a8b-4533-a604-d3478d41989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 29 22:42:24.647768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:24.647725    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:25.536865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:25.536824    4860 patch_prober.go:29] interesting pod/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Oct 29 22:42:25.537043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:25.536891    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9" podUID="46eab5e9-0a8b-4533-a604-d3478d41989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Oct 29 22:42:26.361111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:26.361072    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:26.494818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:26.494765    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:26.544280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:26.544245    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:26.545071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:26.545048    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:26.545818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:26.545783    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:42:26.836527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:26.836493    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:27.339746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:27.339705    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:27.573035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:27.573000    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:28.212443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:28.212377    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:29.885320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:29.885292    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:31.396576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:31.396542    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:31.545873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:31.545842    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:31.867227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:31.865692    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:33.440496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.440462    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:33.458827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.458798    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"]
Oct 29 22:42:33.465078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.465058    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"
Oct 29 22:42:33.482854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.482835    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8147a8\""
Oct 29 22:42:33.500895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.500868    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"]
Oct 29 22:42:33.525100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.525076    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8nz\" (UniqueName: \"kubernetes.io/projected/277117dc-f565-40c3-81d2-8bc9fbd49346-kube-api-access-qn8nz\") pod \"payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk\" (UID: \"277117dc-f565-40c3-81d2-8bc9fbd49346\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"
Oct 29 22:42:33.525258 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.525231    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277117dc-f565-40c3-81d2-8bc9fbd49346-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk\" (UID: \"277117dc-f565-40c3-81d2-8bc9fbd49346\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"
Oct 29 22:42:33.589436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.589399    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"]
Oct 29 22:42:33.596033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.596007    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"
Oct 29 22:42:33.636297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.636252    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8nz\" (UniqueName: \"kubernetes.io/projected/277117dc-f565-40c3-81d2-8bc9fbd49346-kube-api-access-qn8nz\") pod \"payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk\" (UID: \"277117dc-f565-40c3-81d2-8bc9fbd49346\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"
Oct 29 22:42:33.636729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.636703    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277117dc-f565-40c3-81d2-8bc9fbd49346-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk\" (UID: \"277117dc-f565-40c3-81d2-8bc9fbd49346\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"
Oct 29 22:42:33.646352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.637966    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277117dc-f565-40c3-81d2-8bc9fbd49346-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk\" (UID: \"277117dc-f565-40c3-81d2-8bc9fbd49346\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"
Oct 29 22:42:33.659925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.652653    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8147a8-dockercfg-62zdl\""
Oct 29 22:42:33.660742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.660713    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:33.686346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.685793    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"]
Oct 29 22:42:33.690670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.690609    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8nz\" (UniqueName: \"kubernetes.io/projected/277117dc-f565-40c3-81d2-8bc9fbd49346-kube-api-access-qn8nz\") pod \"payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk\" (UID: \"277117dc-f565-40c3-81d2-8bc9fbd49346\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"
Oct 29 22:42:33.737925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.737892    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504b8724-2295-43a1-bf0c-877e006bd0ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2\" (UID: \"504b8724-2295-43a1-bf0c-877e006bd0ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"
Oct 29 22:42:33.738097 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.738003    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpm7b\" (UniqueName: \"kubernetes.io/projected/504b8724-2295-43a1-bf0c-877e006bd0ae-kube-api-access-lpm7b\") pod \"payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2\" (UID: \"504b8724-2295-43a1-bf0c-877e006bd0ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"
Oct 29 22:42:33.780536 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.780503    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"
Oct 29 22:42:33.841713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.840404    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504b8724-2295-43a1-bf0c-877e006bd0ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2\" (UID: \"504b8724-2295-43a1-bf0c-877e006bd0ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"
Oct 29 22:42:33.841713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.840534    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpm7b\" (UniqueName: \"kubernetes.io/projected/504b8724-2295-43a1-bf0c-877e006bd0ae-kube-api-access-lpm7b\") pod \"payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2\" (UID: \"504b8724-2295-43a1-bf0c-877e006bd0ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"
Oct 29 22:42:33.841883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.841736    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504b8724-2295-43a1-bf0c-877e006bd0ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2\" (UID: \"504b8724-2295-43a1-bf0c-877e006bd0ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"
Oct 29 22:42:33.896355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:33.896231    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpm7b\" (UniqueName: \"kubernetes.io/projected/504b8724-2295-43a1-bf0c-877e006bd0ae-kube-api-access-lpm7b\") pod \"payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2\" (UID: \"504b8724-2295-43a1-bf0c-877e006bd0ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"
Oct 29 22:42:34.021932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:34.021833    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"
Oct 29 22:42:34.094413 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:42:34.094377    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277117dc_f565_40c3_81d2_8bc9fbd49346.slice/crio-21f8d982060b5ef2873582598f2a6f5c0364722bb8ce2172adf82c194809f9b4 WatchSource:0}: Error finding container 21f8d982060b5ef2873582598f2a6f5c0364722bb8ce2172adf82c194809f9b4: Status 404 returned error can't find the container with id 21f8d982060b5ef2873582598f2a6f5c0364722bb8ce2172adf82c194809f9b4
Oct 29 22:42:34.097382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:34.097278    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"]
Oct 29 22:42:34.245266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:34.245220    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:34.349820 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:42:34.349777    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504b8724_2295_43a1_bf0c_877e006bd0ae.slice/crio-bf8b1fd978e5ca919f2ae5e9c5d64fb4080d083d14acb4b592511b31d6056040 WatchSource:0}: Error finding container bf8b1fd978e5ca919f2ae5e9c5d64fb4080d083d14acb4b592511b31d6056040: Status 404 returned error can't find the container with id bf8b1fd978e5ca919f2ae5e9c5d64fb4080d083d14acb4b592511b31d6056040
Oct 29 22:42:34.354153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:34.354127    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"]
Oct 29 22:42:34.530245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:34.530210    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk" event={"ID":"277117dc-f565-40c3-81d2-8bc9fbd49346","Type":"ContainerStarted","Data":"784a756d16d2334492c5a09e1dbe585f04e31a85b2a1433a5295f56c1074b8af"}
Oct 29 22:42:34.530673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:34.530261    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk" event={"ID":"277117dc-f565-40c3-81d2-8bc9fbd49346","Type":"ContainerStarted","Data":"21f8d982060b5ef2873582598f2a6f5c0364722bb8ce2172adf82c194809f9b4"}
Oct 29 22:42:34.530673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:34.530290    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"
Oct 29 22:42:34.532059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:34.532035    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2" event={"ID":"504b8724-2295-43a1-bf0c-877e006bd0ae","Type":"ContainerStarted","Data":"fce5b6537c4649c3d61ec68375865556e2ebacb1ad30bd5843d5aaba89b735f5"}
Oct 29 22:42:34.532161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:34.532085    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"
Oct 29 22:42:34.532161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:34.532103    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2" event={"ID":"504b8724-2295-43a1-bf0c-877e006bd0ae","Type":"ContainerStarted","Data":"bf8b1fd978e5ca919f2ae5e9c5d64fb4080d083d14acb4b592511b31d6056040"}
Oct 29 22:42:34.582593 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:34.582553    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk" podStartSLOduration=1.5825413400000001 podStartE2EDuration="1.58254134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:34.580745229 +0000 UTC m=+1696.605521363" watchObservedRunningTime="2026-10-29 22:42:34.58254134 +0000 UTC m=+1696.607317471"
Oct 29 22:42:34.632617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:34.632528    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2" podStartSLOduration=1.6325113 podStartE2EDuration="1.6325113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:34.631516646 +0000 UTC m=+1696.656292777" watchObservedRunningTime="2026-10-29 22:42:34.6325113 +0000 UTC m=+1696.657287432"
Oct 29 22:42:35.984350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:35.980957    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:37.507849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:37.507810    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:37.820452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:37.816946    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:38.000172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.000139    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:38.783942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.783899    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"]
Oct 29 22:42:38.790348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.790310    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:38.811526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.811500    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e619b6-openshift-default-dockercfg-rrqdd\""
Oct 29 22:42:38.833581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.833552    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"]
Oct 29 22:42:38.939990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.939956    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-credential-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:38.940167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.940006    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istiod-ca-cert\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:38.940167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.940091    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-token\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:38.940305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.940196    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-envoy\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:38.940305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.940227    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:38.940305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.940249    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-podinfo\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:38.940555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.940438    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-certs\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:38.940555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.940485    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-data\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:38.940667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.940579    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5829\" (UniqueName: \"kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-kube-api-access-q5829\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:38.993039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:38.992931    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:39.041997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.041907    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-certs\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.041997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.041958    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-data\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.042213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.042025    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5829\" (UniqueName: \"kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-kube-api-access-q5829\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.042213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.042103    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-credential-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.042213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.042139    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istiod-ca-cert\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.042213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.042164    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-token\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.042406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.042258    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-envoy\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.042406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.042293    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.042406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.042334    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-podinfo\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.042406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.042393    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-certs\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.042789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.042768    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.042861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.042844    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-data\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.042920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.042876    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istiod-ca-cert\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.042995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.042973    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-credential-socket\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.044715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.044688    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-envoy\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.044933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.044915    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-podinfo\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.080339 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.080299    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5829\" (UniqueName: \"kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-kube-api-access-q5829\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.082712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.082685    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-token\") pod \"e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") " pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.107257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.107232    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:39.253916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.253882    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:39.439320 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:42:39.439289    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9748a13_8e36_4b49_bc3d_ac909ea8d367.slice/crio-e6de5d18ae6c89c372cea6f1c9570d7fc3af03e023a541f24f512bb3075edbe6 WatchSource:0}: Error finding container e6de5d18ae6c89c372cea6f1c9570d7fc3af03e023a541f24f512bb3075edbe6: Status 404 returned error can't find the container with id e6de5d18ae6c89c372cea6f1c9570d7fc3af03e023a541f24f512bb3075edbe6
Oct 29 22:42:39.444143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.444091    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:39.444294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.444264    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:39.444404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.444384    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:39.447578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.447550    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"]
Oct 29 22:42:39.589601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.589564    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" event={"ID":"a9748a13-8e36-4b49-bc3d-ac909ea8d367","Type":"ContainerStarted","Data":"a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68"}
Oct 29 22:42:39.589785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.589630    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" event={"ID":"a9748a13-8e36-4b49-bc3d-ac909ea8d367","Type":"ContainerStarted","Data":"e6de5d18ae6c89c372cea6f1c9570d7fc3af03e023a541f24f512bb3075edbe6"}
Oct 29 22:42:39.660858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.660775    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" podStartSLOduration=1.6607568860000002 podStartE2EDuration="1.660756886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:39.651217327 +0000 UTC m=+1701.675993531" watchObservedRunningTime="2026-10-29 22:42:39.660756886 +0000 UTC m=+1701.685533020"
Oct 29 22:42:39.881159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:39.881073    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:40.117152 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:40.117125    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:40.117344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:40.117213    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:40.598341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:40.598300    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:40.599409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:40.599386    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:42:41.901774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:41.901742    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:42.129349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:42.124064    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:42.341349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:42.335820    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:43.974348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:43.973967    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:44.269504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.264529    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:44.526638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.526549    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"]
Oct 29 22:42:44.542886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.542848    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:44.590358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.578785    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"
Oct 29 22:42:44.638523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.605222    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"]
Oct 29 22:42:44.652355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.639735    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e619b6\""
Oct 29 22:42:44.716433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.716398    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"]
Oct 29 22:42:44.764965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.746438    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5000257-c412-436d-92f6-9b31c531e94b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm\" (UID: \"a5000257-c412-436d-92f6-9b31c531e94b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"
Oct 29 22:42:44.764965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.763726    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66dsq\" (UniqueName: \"kubernetes.io/projected/a5000257-c412-436d-92f6-9b31c531e94b-kube-api-access-66dsq\") pod \"payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm\" (UID: \"a5000257-c412-436d-92f6-9b31c531e94b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"
Oct 29 22:42:44.764965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.750682    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e619b6-dockercfg-twqrl\""
Oct 29 22:42:44.804430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.795830    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"]
Oct 29 22:42:44.826532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.826502    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"
Oct 29 22:42:44.890349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.872687    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66dsq\" (UniqueName: \"kubernetes.io/projected/a5000257-c412-436d-92f6-9b31c531e94b-kube-api-access-66dsq\") pod \"payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm\" (UID: \"a5000257-c412-436d-92f6-9b31c531e94b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"
Oct 29 22:42:44.891571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.891534    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5000257-c412-436d-92f6-9b31c531e94b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm\" (UID: \"a5000257-c412-436d-92f6-9b31c531e94b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"
Oct 29 22:42:44.893055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.893030    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5000257-c412-436d-92f6-9b31c531e94b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm\" (UID: \"a5000257-c412-436d-92f6-9b31c531e94b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"
Oct 29 22:42:44.997793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.990873    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66dsq\" (UniqueName: \"kubernetes.io/projected/a5000257-c412-436d-92f6-9b31c531e94b-kube-api-access-66dsq\") pod \"payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm\" (UID: \"a5000257-c412-436d-92f6-9b31c531e94b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"
Oct 29 22:42:44.997793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.993435    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6k9z\" (UniqueName: \"kubernetes.io/projected/175c18ae-262c-4dcc-9b92-e0c683837697-kube-api-access-j6k9z\") pod \"payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq\" (UID: \"175c18ae-262c-4dcc-9b92-e0c683837697\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"
Oct 29 22:42:44.997793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:44.993635    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/175c18ae-262c-4dcc-9b92-e0c683837697-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq\" (UID: \"175c18ae-262c-4dcc-9b92-e0c683837697\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"
Oct 29 22:42:45.108349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:45.105880    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6k9z\" (UniqueName: \"kubernetes.io/projected/175c18ae-262c-4dcc-9b92-e0c683837697-kube-api-access-j6k9z\") pod \"payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq\" (UID: \"175c18ae-262c-4dcc-9b92-e0c683837697\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"
Oct 29 22:42:45.108349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:45.106074    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/175c18ae-262c-4dcc-9b92-e0c683837697-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq\" (UID: \"175c18ae-262c-4dcc-9b92-e0c683837697\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"
Oct 29 22:42:45.108349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:45.107150    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/175c18ae-262c-4dcc-9b92-e0c683837697-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq\" (UID: \"175c18ae-262c-4dcc-9b92-e0c683837697\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"
Oct 29 22:42:45.163390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:45.163314    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6k9z\" (UniqueName: \"kubernetes.io/projected/175c18ae-262c-4dcc-9b92-e0c683837697-kube-api-access-j6k9z\") pod \"payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq\" (UID: \"175c18ae-262c-4dcc-9b92-e0c683837697\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"
Oct 29 22:42:45.219522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:45.219488    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"
Oct 29 22:42:45.456370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:45.450286    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"
Oct 29 22:42:45.557756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:45.557732    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"
Oct 29 22:42:45.560197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:45.560178    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"
Oct 29 22:42:45.977348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:45.962312    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:46.361052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:46.361010    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"]
Oct 29 22:42:46.839348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:46.834840    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:47.247193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:47.247133    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:47.262794 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:47.262759    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq" event={"ID":"175c18ae-262c-4dcc-9b92-e0c683837697","Type":"ContainerStarted","Data":"e3f56fd32fcee180de98ca0fb18d1041c02d2f9820764e4531575a1c9cd2baa7"}
Oct 29 22:42:47.263705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:47.263664    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"]
Oct 29 22:42:47.265112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:47.264926    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm" event={"ID":"a5000257-c412-436d-92f6-9b31c531e94b","Type":"ContainerStarted","Data":"01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15"}
Oct 29 22:42:47.265112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:47.264969    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm" event={"ID":"a5000257-c412-436d-92f6-9b31c531e94b","Type":"ContainerStarted","Data":"8fe64901c343299ccafa1a7c78d38695a134121ec8a68e0dd23c0566a03802a7"}
Oct 29 22:42:47.265376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:47.265356    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"
Oct 29 22:42:47.365731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:47.365677    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm" podStartSLOduration=3.365657396 podStartE2EDuration="3.365657396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:47.331155049 +0000 UTC m=+1709.355931198" watchObservedRunningTime="2026-10-29 22:42:47.365657396 +0000 UTC m=+1709.390433529"
Oct 29 22:42:47.660961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:47.660860    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:47.830398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:47.813798    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:47.986103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:47.985940    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:48.221848 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:42:48.221710    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:42:48.337361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:48.334676    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq" event={"ID":"175c18ae-262c-4dcc-9b92-e0c683837697","Type":"ContainerStarted","Data":"dc0f80074b668b5c077bb907e34606311950c7ff1d99164bafda28a0a6587c40"}
Oct 29 22:42:48.337361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:48.334766    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"
Oct 29 22:42:48.388348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:48.388268    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq" podStartSLOduration=4.388247758 podStartE2EDuration="4.388247758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:48.387674473 +0000 UTC m=+1710.412450605" watchObservedRunningTime="2026-10-29 22:42:48.388247758 +0000 UTC m=+1710.413023890"
Oct 29 22:42:49.425320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:49.425231    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:51.168616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:51.168530    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:51.804037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:51.804000    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:51.954670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:51.954633    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:52.952885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:52.952843    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:53.914904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:53.914862    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:54.221392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:54.220646    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:54.981620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:54.980915    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:55.342833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:55.333698    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:56.782966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.741801    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:56.782966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.752859    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"]
Oct 29 22:42:56.782966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.771955    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"]
Oct 29 22:42:56.782966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.772119    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:56.866280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.866213    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:56.866280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.866247    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff9440f5-5b2c-4695-b1b1-366c233a7187-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:56.866280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.866269    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.866447    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.866482    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.866562    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.866685    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvxb\" (UniqueName: \"kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-kube-api-access-4gvxb\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.866758    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.866795    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.967992    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.968023    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff9440f5-5b2c-4695-b1b1-366c233a7187-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.968043    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.968091    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.968122    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.968198    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.968314    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvxb\" (UniqueName: \"kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-kube-api-access-4gvxb\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.968379    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.968431    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.968447    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-credential-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.968525    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-data\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.968788    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-certs\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.968862    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff9440f5-5b2c-4695-b1b1-366c233a7187-istiod-ca-cert\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.969014    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-socket\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.970792    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-envoy\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.970897    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-podinfo\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.990821    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvxb\" (UniqueName: \"kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-kube-api-access-4gvxb\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:56.994011    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-token\") pod \"e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") " pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.151157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.119764    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:57.432093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.432065    4860 generic.go:356] "Generic (PLEG): container finished" podID="c8790a27-ef61-43db-803b-fc295959b33a" containerID="8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72" exitCode=0
Oct 29 22:42:57.432281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.432141    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" event={"ID":"c8790a27-ef61-43db-803b-fc295959b33a","Type":"ContainerDied","Data":"8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72"}
Oct 29 22:42:57.432721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.432702    4860 scope.go:122] "RemoveContainer" containerID="8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72"
Oct 29 22:42:57.434697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.434673    4860 generic.go:356] "Generic (PLEG): container finished" podID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerID="a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de" exitCode=0
Oct 29 22:42:57.434804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.434745    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" event={"ID":"98bb03a1-504e-4c1e-8df1-d6af23cd10ed","Type":"ContainerDied","Data":"a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de"}
Oct 29 22:42:57.435162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.435143    4860 scope.go:122] "RemoveContainer" containerID="a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de"
Oct 29 22:42:57.435246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.435134    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:57.435312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.435300    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:57.435386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.435372    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:57.459227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.459199    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:57.459384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.459279    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:57.459384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.459360    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:57.479445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.479421    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"]
Oct 29 22:42:57.488710 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:42:57.488682    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9440f5_5b2c_4695_b1b1_366c233a7187.slice/crio-9226d0c812688d3613dddbbc35b9aac5eb04e6c40b0429122257e672db0c52ae WatchSource:0}: Error finding container 9226d0c812688d3613dddbbc35b9aac5eb04e6c40b0429122257e672db0c52ae: Status 404 returned error can't find the container with id 9226d0c812688d3613dddbbc35b9aac5eb04e6c40b0429122257e672db0c52ae
Oct 29 22:42:57.492285 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.492250    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:57.492432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.492344    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:57.492432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:57.492396    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:58.353011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.352812    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"
Oct 29 22:42:58.439207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.439063    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:42:58.464356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.464301    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" event={"ID":"ff9440f5-5b2c-4695-b1b1-366c233a7187","Type":"ContainerStarted","Data":"6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e"}
Oct 29 22:42:58.464621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.464592    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" event={"ID":"ff9440f5-5b2c-4695-b1b1-366c233a7187","Type":"ContainerStarted","Data":"9226d0c812688d3613dddbbc35b9aac5eb04e6c40b0429122257e672db0c52ae"}
Oct 29 22:42:58.508770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.508723    4860 generic.go:356] "Generic (PLEG): container finished" podID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerID="ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac" exitCode=0
Oct 29 22:42:58.509163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.509134    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" event={"ID":"dffef942-b70f-496b-9dd1-aa8921e066ad","Type":"ContainerDied","Data":"ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac"}
Oct 29 22:42:58.525380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.517994    4860 scope.go:122] "RemoveContainer" containerID="ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac"
Oct 29 22:42:58.538260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.538214    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:58.540273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.540226    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:58.540598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.540565    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:42:58.544305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.539648    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:58.653566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.542816    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" event={"ID":"c8790a27-ef61-43db-803b-fc295959b33a","Type":"ContainerStarted","Data":"7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646"}
Oct 29 22:42:58.767780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.767746    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" event={"ID":"98bb03a1-504e-4c1e-8df1-d6af23cd10ed","Type":"ContainerStarted","Data":"9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4"}
Oct 29 22:42:58.805374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.804943    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" podStartSLOduration=2.804918932 podStartE2EDuration="2.804918932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:58.646015521 +0000 UTC m=+1720.670791643" watchObservedRunningTime="2026-10-29 22:42:58.804918932 +0000 UTC m=+1720.829695063"
Oct 29 22:42:58.811522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.811484    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:42:58.858153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.834728    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:42:58.858153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:58.834789    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 29 22:42:59.127446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:59.123143    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:42:59.127446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:59.123261    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:42:59.127446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:59.123302    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 29 22:42:59.441144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:59.435737    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:42:59.441144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:59.436876    4860 patch_prober.go:29] interesting pod/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 29 22:42:59.441144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:59.436941    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 29 22:42:59.664430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:59.664026    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:42:59.815343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:59.814804    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:42:59.815343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:59.814882    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 29 22:42:59.865298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:42:59.865267    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" event={"ID":"dffef942-b70f-496b-9dd1-aa8921e066ad","Type":"ContainerStarted","Data":"627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc"}
Oct 29 22:43:00.132360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.131949    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:43:00.132360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.132005    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 29 22:43:00.344375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.341193    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:00.372123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.363756    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"
Oct 29 22:43:00.432726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.431970    4860 patch_prober.go:29] interesting pod/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 29 22:43:00.432726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.432049    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 29 22:43:00.702690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.702610    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:43:00.723039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.703273    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 29 22:43:00.723306 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.723282    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 29 22:43:00.742228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.722816    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:00.815450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.815407    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:43:00.826585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.826506    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 29 22:43:00.895724 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:00.888014    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:43:01.127410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:01.126683    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:43:01.127410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:01.126741    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 29 22:43:01.180347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:01.168506    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:01.409404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:01.408719    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:01.437355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:01.435342    4860 patch_prober.go:29] interesting pod/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 29 22:43:01.437355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:01.435402    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 29 22:43:01.638349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:01.627779    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:01.802240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:01.790775    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main" probeResult="failure" output="Get \"https://10.132.0.154:8000/ready\": context deadline exceeded"
Oct 29 22:43:01.802240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:01.798177    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 29 22:43:01.802240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:01.798234    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 29 22:43:01.920355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:01.918953    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:43:01.920355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:01.919017    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 29 22:43:02.187350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.186842    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:43:02.187350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.187289    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Oct 29 22:43:02.271589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.252716    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:02.304189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.293424    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:43:02.321128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.321081    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:02.438403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.420100    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:43:02.438403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.429665    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 29 22:43:02.438403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.429439    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:02.464348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.463661    4860 patch_prober.go:29] interesting pod/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 29 22:43:02.464348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.463728    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 29 22:43:02.580545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.580506    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"]
Oct 29 22:43:02.622337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.616144    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"]
Oct 29 22:43:02.622337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.616197    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"]
Oct 29 22:43:02.622337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.616221    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b0d2c5-f5l72"]
Oct 29 22:43:02.622337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.617433    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.626694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.626661    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:02.636464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.636429    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:02.648393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.638309    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:43:02.665095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.653429    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-envoy\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.665095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.653544    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-podinfo\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.665095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.653578    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-token\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.665095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.653684    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-credential-socket\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.665095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.653743    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c897948d-a9ba-4e67-81c5-d0830ac823cd-istiod-ca-cert\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.665095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.653773    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-certs\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.665095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.653875    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-data\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.665095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.653999    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkxrq\" (UniqueName: \"kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-kube-api-access-qkxrq\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.665095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.654159    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-socket\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.676725 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.676624    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-05417d-openshift-default-dockercfg-6shxw\""
Oct 29 22:43:02.725345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.725160    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 29 22:43:02.725741 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.725710    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 29 22:43:02.762628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.759372    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-data\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.762628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.759491    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkxrq\" (UniqueName: \"kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-kube-api-access-qkxrq\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.762628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.759604    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-socket\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.762628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.759687    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-envoy\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.762628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.759801    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-podinfo\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.762628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.759830    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-token\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.762628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.759912    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-credential-socket\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.762628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.759969    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c897948d-a9ba-4e67-81c5-d0830ac823cd-istiod-ca-cert\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.762628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.760001    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-certs\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.762628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.760568    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-certs\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.762628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.761812    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-data\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.762628 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.762301    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-socket\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.770470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.767266    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-credential-socket\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.770470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.767977    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c897948d-a9ba-4e67-81c5-d0830ac823cd-istiod-ca-cert\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.795731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.792567    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:02.798186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.798154    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:43:02.798536 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.798511    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 29 22:43:02.805396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.805373    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-envoy\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.848354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.847756    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-podinfo\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.880488 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.860958    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:02.916844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.912554    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-token\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:02.943281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.943237    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:43:02.953228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:02.953181    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:03.113404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.105956    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:43:03.192359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.189645    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:43:03.192359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.189703    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 29 22:43:03.230262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.230196    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkxrq\" (UniqueName: \"kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-kube-api-access-qkxrq\") pod \"e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") " pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:03.263388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.262066    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:03.309922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.304594    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:03.445454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.429756    4860 patch_prober.go:29] interesting pod/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 29 22:43:03.445454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.429814    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 29 22:43:03.554422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.554380    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"]
Oct 29 22:43:03.663482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.661736    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9b0e545-smzcb"]
Oct 29 22:43:03.722535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.722439    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 29 22:43:03.722797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.722769    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 29 22:43:03.797180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.796736    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:43:03.797180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.796794    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 29 22:43:03.967348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.954057    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:03.981063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:03.967831    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:43:04.023410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.022615    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:04.124383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.124348    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:43:04.124645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.124609    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 29 22:43:04.268063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.267551    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:43:04.275370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.271465    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:04.275370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.271774    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:43:04.275370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.272033    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:43:04.432351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.323903    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:04.460203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.444933    4860 patch_prober.go:29] interesting pod/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Oct 29 22:43:04.460203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.444989    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Oct 29 22:43:04.508240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.508204    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1818f73-498f-47fd-afab-467d76cf7209" path="/var/lib/kubelet/pods/d1818f73-498f-47fd-afab-467d76cf7209/volumes"
Oct 29 22:43:04.527421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.509243    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8da5d76-1168-40e9-97ae-b320a9b84c76" path="/var/lib/kubelet/pods/f8da5d76-1168-40e9-97ae-b320a9b84c76/volumes"
Oct 29 22:43:04.529435 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.528584    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:04.572518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.571071    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main" probeResult="failure" output="Get \"https://10.132.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:04.704605 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.704487    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 29 22:43:04.704605 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.704544    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 29 22:43:04.760598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.760550    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:04.797219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.797114    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:43:04.994469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.994412    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 29 22:43:04.995056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.985640    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:04.995532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:04.984797    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:05.021252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.013011    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:05.021848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.013060    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:43:05.039855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.039806    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:05.248348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.247464    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 29 22:43:05.248348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.247551    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 29 22:43:05.331577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.276095    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:05.418777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.416165    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:43:05.418777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.416424    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:43:05.418777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.416486    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:43:05.434653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.434546    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:05.435743 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.435674    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": context deadline exceeded"
Oct 29 22:43:05.436439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.436352    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:43:05.437129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.437043    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:43:05.437932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.437874    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:43:05.442506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.442468    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:43:05.442858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.442809    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:43:05.443149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.443121    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:43:05.490552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.485819    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:43:05.490552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.488596    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:43:05.490552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.488682    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:43:05.566786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.566752    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:43:05.569396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.567707    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:43:05.596391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.596282    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"]
Oct 29 22:43:05.597020 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.596977    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:05.598400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.598382    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:43:05.653832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.653789    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:43:05.676768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.676736    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:43:05.681362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.677740    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main" probeResult="failure" output="Get \"https://10.132.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:43:05.692559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.681650    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:43:05.701868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.701626    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 29 22:43:05.701868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.701688    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 29 22:43:05.793650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.793601    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:43:05.798083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.798062    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:43:05.804888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.804867    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:43:05.813060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.813039    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:43:05.815216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.815195    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:43:05.944087 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:05.944026    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:43:06.234990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:06.234928    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:43:06.237197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:06.237176    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:43:06.244424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:06.244396    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:43:06.262029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:06.260118    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" event={"ID":"c897948d-a9ba-4e67-81c5-d0830ac823cd","Type":"ContainerStarted","Data":"e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f"}
Oct 29 22:43:06.266812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:06.266479    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" event={"ID":"c897948d-a9ba-4e67-81c5-d0830ac823cd","Type":"ContainerStarted","Data":"d8991f6e437b07fb3680eda82579c04f55930ec5b3c22c7617986b3981381d15"}
Oct 29 22:43:06.314656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:06.314558    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:06.357161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:06.357113    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:43:06.358896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:06.358825    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 22:43:06.614738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:06.614646    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:06.702403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:06.702360    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 29 22:43:06.702587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:06.702512    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 29 22:43:06.719658 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:06.719609    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" podStartSLOduration=5.719592263 podStartE2EDuration="5.719592263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:06.718574376 +0000 UTC m=+1728.743350539" watchObservedRunningTime="2026-10-29 22:43:06.719592263 +0000 UTC m=+1728.744368396"
Oct 29 22:43:07.312174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:07.312146    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:07.701637 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:07.701603    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 29 22:43:07.701823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:07.701657    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 29 22:43:08.303491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:08.303453    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:08.304394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:08.304373    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:43:08.719946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:08.719922    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:08.784570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:08.784542    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:43:08.785142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:08.785118    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:43:08.786042 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:08.786021    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:43:09.089281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:09.089200    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:09.277679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:09.277650    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:09.673404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:09.673374    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:09.931091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:09.931059    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:10.958389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:10.957982    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:11.212211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.212129    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:11.277977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.277946    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"]
Oct 29 22:43:11.283412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.283383    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"
Oct 29 22:43:11.304170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.304141    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"]
Oct 29 22:43:11.304806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.304786    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-05417d\""
Oct 29 22:43:11.362304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.362268    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvtwr\" (UniqueName: \"kubernetes.io/projected/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-kube-api-access-gvtwr\") pod \"payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv\" (UID: \"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"
Oct 29 22:43:11.362485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.362466    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv\" (UID: \"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"
Oct 29 22:43:11.489400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.488470    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv\" (UID: \"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"
Oct 29 22:43:11.489400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.488836    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvtwr\" (UniqueName: \"kubernetes.io/projected/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-kube-api-access-gvtwr\") pod \"payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv\" (UID: \"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"
Oct 29 22:43:11.502410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.490149    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv\" (UID: \"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"
Oct 29 22:43:11.506900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.506860    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:11.621545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.617371    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvtwr\" (UniqueName: \"kubernetes.io/projected/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-kube-api-access-gvtwr\") pod \"payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv\" (UID: \"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"
Oct 29 22:43:11.899742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:11.899665    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"
Oct 29 22:43:12.215281 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:43:12.215243    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33354fa_3db1_4e3c_8cf3_65ff12a69ef4.slice/crio-d2a101e1f306008d29c46a78cf892113bad0995aad22604d1cb6a3091e3e4a1a WatchSource:0}: Error finding container d2a101e1f306008d29c46a78cf892113bad0995aad22604d1cb6a3091e3e4a1a: Status 404 returned error can't find the container with id d2a101e1f306008d29c46a78cf892113bad0995aad22604d1cb6a3091e3e4a1a
Oct 29 22:43:12.217770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:12.217735    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"]
Oct 29 22:43:12.351207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:12.351171    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv" event={"ID":"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4","Type":"ContainerStarted","Data":"2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682"}
Oct 29 22:43:12.351361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:12.351234    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv" event={"ID":"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4","Type":"ContainerStarted","Data":"d2a101e1f306008d29c46a78cf892113bad0995aad22604d1cb6a3091e3e4a1a"}
Oct 29 22:43:12.351419 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:12.351362    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"
Oct 29 22:43:12.402071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:12.402018    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv" podStartSLOduration=1.402000043 podStartE2EDuration="1.402000043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:12.401275521 +0000 UTC m=+1734.426051652" watchObservedRunningTime="2026-10-29 22:43:12.402000043 +0000 UTC m=+1734.426776173"
Oct 29 22:43:12.568891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:12.568708    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:12.859139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:12.859057    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:13.030188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:13.030155    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:13.225073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:13.225041    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:14.235101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:14.235070    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:14.433101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:14.433070    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:15.392780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:15.392746    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:17.267266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:17.267237    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:18.058110 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:43:18.058072    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:43:19.036571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:19.036528    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:20.023507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:20.023476    4860 scope.go:122] "RemoveContainer" containerID="dd8eb39284069a1d964619c2021379a401fe5a5a245ac49d963be2da27c1ab21"
Oct 29 22:43:20.832506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:20.832468    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:22.554905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:22.554869    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:23.363412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:23.363385    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"
Oct 29 22:43:24.265424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:24.265387    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:24.421774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:24.421741    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:24.752822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:24.752792    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:26.406215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:26.406181    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:28.127294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:28.127235    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:29.957913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:29.957828    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:30.674813 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:30.674771    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:30.874539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:30.874479    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:31.656619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:31.656552    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:33.333718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:33.333685    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:35.037363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:35.037295    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:35.184432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:35.183413    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:37.043381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:37.043310    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:38.776425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:38.776386    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:40.426036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:40.425999    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:41.410290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:41.410258    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:41.589231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:41.589195    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:42.154101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:42.154062    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:43.880710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:43.880659    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:45.584875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:45.584840    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:47.455185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:47.455089    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:48.058599 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:43:48.058562    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:43:49.317314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:49.317278    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:51.012111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:51.012073    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:52.810888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:52.810851    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:52.933875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:52.933835    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:53.069771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:53.069671    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:54.826268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:54.823773    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:54.958539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:54.958504    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:56.705545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:56.705499    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:57.489531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.489499    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"]
Oct 29 22:43:57.494743 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.494727    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:43:57.517424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.517399    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"]
Oct 29 22:43:57.523679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.523657    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:43:57.523804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.523691    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:43:57.574420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.574390    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3234dde-rvgp9\" (UID: \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:43:57.574606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.574584    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3234dde-rvgp9\" (UID: \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:43:57.574783 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.574763    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fn2h\" (UniqueName: \"kubernetes.io/projected/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-kube-api-access-9fn2h\") pod \"maas-api-revoke-sub-e3234dde-rvgp9\" (UID: \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:43:57.676248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.676222    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3234dde-rvgp9\" (UID: \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:43:57.676408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.676319    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3234dde-rvgp9\" (UID: \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:43:57.676477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.676453    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fn2h\" (UniqueName: \"kubernetes.io/projected/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-kube-api-access-9fn2h\") pod \"maas-api-revoke-sub-e3234dde-rvgp9\" (UID: \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:43:57.676977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.676954    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3234dde-rvgp9\" (UID: \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:43:57.678829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.678804    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3234dde-rvgp9\" (UID: \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:43:57.698697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.698672    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fn2h\" (UniqueName: \"kubernetes.io/projected/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-kube-api-access-9fn2h\") pod \"maas-api-revoke-sub-e3234dde-rvgp9\" (UID: \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:43:57.809986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:57.809930    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:43:58.133507 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:43:58.133469    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7ba7cd_6f95_44bc_b8d7_5bb61e6d8ee2.slice/crio-a4186336fe04d42aeb7981afcda0e3b70163fef4bf22006122fa9fc295d461aa WatchSource:0}: Error finding container a4186336fe04d42aeb7981afcda0e3b70163fef4bf22006122fa9fc295d461aa: Status 404 returned error can't find the container with id a4186336fe04d42aeb7981afcda0e3b70163fef4bf22006122fa9fc295d461aa
Oct 29 22:43:58.135079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:58.135048    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"]
Oct 29 22:43:58.717172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:58.706479    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:58.916511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:58.916299    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9" event={"ID":"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2","Type":"ContainerStarted","Data":"f3805db84d8c0ebfabf301a304c3b4c30b776720c41e652197590400594507c5"}
Oct 29 22:43:58.916511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:58.916405    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9" event={"ID":"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2","Type":"ContainerStarted","Data":"a4186336fe04d42aeb7981afcda0e3b70163fef4bf22006122fa9fc295d461aa"}
Oct 29 22:43:58.972976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:58.972888    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9" podStartSLOduration=1.9728749859999999 podStartE2EDuration="1.972874986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:58.968271238 +0000 UTC m=+1780.993047394" watchObservedRunningTime="2026-10-29 22:43:58.972874986 +0000 UTC m=+1780.997651117"
Oct 29 22:43:59.929345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:59.929298    4860 generic.go:356] "Generic (PLEG): container finished" podID="cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2" containerID="f3805db84d8c0ebfabf301a304c3b4c30b776720c41e652197590400594507c5" exitCode=0
Oct 29 22:43:59.929775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:43:59.929392    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9" event={"ID":"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2","Type":"ContainerDied","Data":"f3805db84d8c0ebfabf301a304c3b4c30b776720c41e652197590400594507c5"}
Oct 29 22:44:00.414952 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:00.410250    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:01.131723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.131700    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:44:01.148609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.148584    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-xks-ca\") pod \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\" (UID: \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\") "
Oct 29 22:44:01.148752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.148684    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-openshift-ca\") pod \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\" (UID: \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\") "
Oct 29 22:44:01.148752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.148733    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-kube-api-access-9fn2h\" (UniqueName: \"kubernetes.io/projected/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-kube-api-access-9fn2h\") pod \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\" (UID: \"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2\") "
Oct 29 22:44:01.149175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.149135    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-openshift-ca" pod "cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2" (UID: "cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:44:01.149618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.149593    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:44:01.151338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.151302    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-xks-ca" pod "cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2" (UID: "cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:44:01.151431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.151404    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-kube-api-access-9fn2h" pod "cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2" (UID: "cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2"). InnerVolumeSpecName "kube-api-access-9fn2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:01.251459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.251382    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fn2h\" (UniqueName: \"kubernetes.io/projected/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-kube-api-access-9fn2h\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:44:01.251459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.251409    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:44:01.992237 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.992198    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9" event={"ID":"cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2","Type":"ContainerDied","Data":"a4186336fe04d42aeb7981afcda0e3b70163fef4bf22006122fa9fc295d461aa"}
Oct 29 22:44:01.992540 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.992517    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4186336fe04d42aeb7981afcda0e3b70163fef4bf22006122fa9fc295d461aa"
Oct 29 22:44:01.992649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:01.992247    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"
Oct 29 22:44:02.248031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:02.247799    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:03.588930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:03.588883    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:04.522550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:04.522520    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:05.242458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:05.242428    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:07.155175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:07.155141    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:10.118487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:10.118456    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:11.489474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:11.489436    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:12.712227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:12.712186    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:13.550381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:13.550349    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:15.284557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:15.284523    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:17.131959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:17.131922    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:18.057452 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:44:18.057415    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:44:18.431654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:18.431621    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:44:18.435936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:18.435907    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:44:18.809404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:18.809302    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:19.431275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.431240    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:19.623678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.623643    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:19.835428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.835339    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"]
Oct 29 22:44:19.836417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.836384    4860 cpu_manager.go:414] "containerMap: removing container" podUID="cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2" containerName="revoke-keys"
Oct 29 22:44:19.836573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.836420    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2" containerName="revoke-keys"
Oct 29 22:44:19.836745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.836731    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2" containerName="revoke-keys"
Oct 29 22:44:19.841642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.841619    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:19.857091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.857063    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"]
Oct 29 22:44:19.864113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.864088    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:44:19.875881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.875857    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:44:19.930169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.930131    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-528518e3-w8fqv\" (UID: \"50578f72-af6a-4e68-88a8-8854f88ee53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:19.930351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.930197    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-528518e3-w8fqv\" (UID: \"50578f72-af6a-4e68-88a8-8854f88ee53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:19.930351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:19.930333    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t58l\" (UniqueName: \"kubernetes.io/projected/50578f72-af6a-4e68-88a8-8854f88ee53f-kube-api-access-2t58l\") pod \"maas-api-revoke-sub-528518e3-w8fqv\" (UID: \"50578f72-af6a-4e68-88a8-8854f88ee53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:20.032180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.032130    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-528518e3-w8fqv\" (UID: \"50578f72-af6a-4e68-88a8-8854f88ee53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:20.032388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.032208    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-528518e3-w8fqv\" (UID: \"50578f72-af6a-4e68-88a8-8854f88ee53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:20.032482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.032459    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t58l\" (UniqueName: \"kubernetes.io/projected/50578f72-af6a-4e68-88a8-8854f88ee53f-kube-api-access-2t58l\") pod \"maas-api-revoke-sub-528518e3-w8fqv\" (UID: \"50578f72-af6a-4e68-88a8-8854f88ee53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:20.032974 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.032949    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-528518e3-w8fqv\" (UID: \"50578f72-af6a-4e68-88a8-8854f88ee53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:20.034785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.034761    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-528518e3-w8fqv\" (UID: \"50578f72-af6a-4e68-88a8-8854f88ee53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:20.057280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.057242    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t58l\" (UniqueName: \"kubernetes.io/projected/50578f72-af6a-4e68-88a8-8854f88ee53f-kube-api-access-2t58l\") pod \"maas-api-revoke-sub-528518e3-w8fqv\" (UID: \"50578f72-af6a-4e68-88a8-8854f88ee53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:20.057889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.057868    4860 scope.go:122] "RemoveContainer" containerID="b1f7e9c88dde03684d6d703dae779a1518e56ed22302cab59aefd2515d7c880e"
Oct 29 22:44:20.086396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.086367    4860 scope.go:122] "RemoveContainer" containerID="5b460a85fd0cb4276ed7231b682c84162acbe92a05d7fbf4aca1b027f31a9f5c"
Oct 29 22:44:20.100405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.100377    4860 scope.go:122] "RemoveContainer" containerID="f11630547b811653b0184291134d22b10542abbc3713dfc44d12d654e16e0253"
Oct 29 22:44:20.112226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.112206    4860 scope.go:122] "RemoveContainer" containerID="f0073e6d711eb14b5faa607b88693a84631695fa03ded98cb6f7f6d2f1822350"
Oct 29 22:44:20.123409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.123382    4860 scope.go:122] "RemoveContainer" containerID="bdde5a5ce63bbdd750d40f8d2904a598a72157f4e04fe1f6c2e37a00d4c93858"
Oct 29 22:44:20.163954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.163927    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:20.714861 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:44:20.714831    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50578f72_af6a_4e68_88a8_8854f88ee53f.slice/crio-48beb875440bd01292616b83263ff34733027b1cbff6a60c58bcba50a98bc163 WatchSource:0}: Error finding container 48beb875440bd01292616b83263ff34733027b1cbff6a60c58bcba50a98bc163: Status 404 returned error can't find the container with id 48beb875440bd01292616b83263ff34733027b1cbff6a60c58bcba50a98bc163
Oct 29 22:44:20.717537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:20.717510    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"]
Oct 29 22:44:21.206790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:21.206751    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:21.320347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:21.317585    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv" event={"ID":"50578f72-af6a-4e68-88a8-8854f88ee53f","Type":"ContainerStarted","Data":"48beb875440bd01292616b83263ff34733027b1cbff6a60c58bcba50a98bc163"}
Oct 29 22:44:22.336377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:22.334672    4860 generic.go:356] "Generic (PLEG): container finished" podID="50578f72-af6a-4e68-88a8-8854f88ee53f" containerID="091f5ceb9a43ac10c55c602762ed98133aa7aef3f9b755820d810f5898ceb307" exitCode=0
Oct 29 22:44:22.336377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:22.334726    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv" event={"ID":"50578f72-af6a-4e68-88a8-8854f88ee53f","Type":"ContainerDied","Data":"091f5ceb9a43ac10c55c602762ed98133aa7aef3f9b755820d810f5898ceb307"}
Oct 29 22:44:22.861483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:22.861453    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:23.665470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:23.665435    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:23.741421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:23.741384    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-openshift-ca\") pod \"50578f72-af6a-4e68-88a8-8854f88ee53f\" (UID: \"50578f72-af6a-4e68-88a8-8854f88ee53f\") "
Oct 29 22:44:23.741614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:23.741543    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-xks-ca\") pod \"50578f72-af6a-4e68-88a8-8854f88ee53f\" (UID: \"50578f72-af6a-4e68-88a8-8854f88ee53f\") "
Oct 29 22:44:23.741614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:23.741600    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50578f72-af6a-4e68-88a8-8854f88ee53f-kube-api-access-2t58l\" (UniqueName: \"kubernetes.io/projected/50578f72-af6a-4e68-88a8-8854f88ee53f-kube-api-access-2t58l\") pod \"50578f72-af6a-4e68-88a8-8854f88ee53f\" (UID: \"50578f72-af6a-4e68-88a8-8854f88ee53f\") "
Oct 29 22:44:23.743252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:23.743222    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-openshift-ca" pod "50578f72-af6a-4e68-88a8-8854f88ee53f" (UID: "50578f72-af6a-4e68-88a8-8854f88ee53f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:44:23.745433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:23.745408    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50578f72-af6a-4e68-88a8-8854f88ee53f-kube-api-access-2t58l" pod "50578f72-af6a-4e68-88a8-8854f88ee53f" (UID: "50578f72-af6a-4e68-88a8-8854f88ee53f"). InnerVolumeSpecName "kube-api-access-2t58l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:23.808375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:23.780758    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-xks-ca" pod "50578f72-af6a-4e68-88a8-8854f88ee53f" (UID: "50578f72-af6a-4e68-88a8-8854f88ee53f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:44:23.844973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:23.844913    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:44:23.844973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:23.844967    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50578f72-af6a-4e68-88a8-8854f88ee53f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:44:23.844973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:23.844986    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t58l\" (UniqueName: \"kubernetes.io/projected/50578f72-af6a-4e68-88a8-8854f88ee53f-kube-api-access-2t58l\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:44:24.353910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:24.353876    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"
Oct 29 22:44:24.354085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:24.353910    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv" event={"ID":"50578f72-af6a-4e68-88a8-8854f88ee53f","Type":"ContainerDied","Data":"48beb875440bd01292616b83263ff34733027b1cbff6a60c58bcba50a98bc163"}
Oct 29 22:44:24.354085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:24.353992    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48beb875440bd01292616b83263ff34733027b1cbff6a60c58bcba50a98bc163"
Oct 29 22:44:24.759283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:24.759088    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:26.551953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:26.551904    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:28.460584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:28.460545    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:29.097860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:29.097775    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:30.304148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:30.304114    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:32.094773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:32.093294    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:33.514921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:33.514889    4860 generic.go:356] "Generic (PLEG): container finished" podID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerID="a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68" exitCode=0
Oct 29 22:44:33.515320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:33.514965    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" event={"ID":"a9748a13-8e36-4b49-bc3d-ac909ea8d367","Type":"ContainerDied","Data":"a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68"}
Oct 29 22:44:33.515522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:33.515500    4860 scope.go:122] "RemoveContainer" containerID="a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68"
Oct 29 22:44:33.516863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:33.516841    4860 generic.go:356] "Generic (PLEG): container finished" podID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerID="e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f" exitCode=0
Oct 29 22:44:33.516947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:33.516922    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" event={"ID":"c897948d-a9ba-4e67-81c5-d0830ac823cd","Type":"ContainerDied","Data":"e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f"}
Oct 29 22:44:33.517400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:33.517383    4860 scope.go:122] "RemoveContainer" containerID="e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f"
Oct 29 22:44:33.517823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:33.517790    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:44:33.517898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:33.517882    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:44:33.517948 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:33.517921    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:44:33.519254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:33.519228    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:44:33.519311 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:33.519294    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:44:33.519391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:33.519340    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:44:34.093520 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:34.093486    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:34.535204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:34.535160    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" event={"ID":"c897948d-a9ba-4e67-81c5-d0830ac823cd","Type":"ContainerStarted","Data":"b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49"}
Oct 29 22:44:34.542350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:34.539368    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" event={"ID":"a9748a13-8e36-4b49-bc3d-ac909ea8d367","Type":"ContainerStarted","Data":"b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce"}
Oct 29 22:44:35.110349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:35.109740    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:44:35.110349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:35.109866    4860 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Oct 29 22:44:35.110349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:35.109926    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Oct 29 22:44:35.317398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:35.311705    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:44:35.317398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:35.315429    4860 patch_prober.go:29] interesting pod/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 29 22:44:35.317398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:35.315490    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 29 22:44:35.831348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:35.820947    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:36.119589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:36.119486    4860 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Oct 29 22:44:36.119899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:36.119868    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Oct 29 22:44:36.307349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:36.306675    4860 patch_prober.go:29] interesting pod/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 29 22:44:36.307349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:36.306736    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 29 22:44:37.111234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:37.111198    4860 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Oct 29 22:44:37.121813 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:37.121773    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Oct 29 22:44:37.312349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:37.312193    4860 patch_prober.go:29] interesting pod/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 29 22:44:37.312349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:37.312255    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 29 22:44:37.502346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:37.502173    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:38.122352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:38.122151    4860 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Oct 29 22:44:38.122352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:38.122208    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Oct 29 22:44:38.245715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:38.245671    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:38.323504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:38.323463    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:44:38.324958 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:38.324924    4860 patch_prober.go:29] interesting pod/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 29 22:44:38.325116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:38.324981    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 29 22:44:39.099497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:39.099457    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:39.107809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:39.107773    4860 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Oct 29 22:44:39.108083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:39.108051    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Oct 29 22:44:39.323052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:39.323011    4860 patch_prober.go:29] interesting pod/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 29 22:44:39.323738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:39.323687    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 29 22:44:40.108518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:40.108481    4860 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Oct 29 22:44:40.108747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:40.108536    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Oct 29 22:44:40.342387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:40.341247    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:44:40.346231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:40.346202    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:44:40.598397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:40.598359    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:44:40.754226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:40.754189    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:41.112703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:41.112672    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:44:41.113694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:41.113675    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:44:42.458663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:42.458628    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:43.262843 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.262808    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:43.503469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.503429    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"]
Oct 29 22:44:43.504768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.504744    4860 cpu_manager.go:414] "containerMap: removing container" podUID="50578f72-af6a-4e68-88a8-8854f88ee53f" containerName="revoke-keys"
Oct 29 22:44:43.504768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.504766    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50578f72-af6a-4e68-88a8-8854f88ee53f" containerName="revoke-keys"
Oct 29 22:44:43.504967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.504938    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="50578f72-af6a-4e68-88a8-8854f88ee53f" containerName="revoke-keys"
Oct 29 22:44:43.509173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.509067    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:43.528841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.528812    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f835847-daae-471e-a710-84ee468eff27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p\" (UID: \"5f835847-daae-471e-a710-84ee468eff27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:43.529010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.528988    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f835847-daae-471e-a710-84ee468eff27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p\" (UID: \"5f835847-daae-471e-a710-84ee468eff27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:43.529320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.529280    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxpj9\" (UniqueName: \"kubernetes.io/projected/5f835847-daae-471e-a710-84ee468eff27-kube-api-access-fxpj9\") pod \"maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p\" (UID: \"5f835847-daae-471e-a710-84ee468eff27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:43.540543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.540518    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"]
Oct 29 22:44:43.541992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.541963    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:44:43.542095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.542045    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:44:43.630917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.630885    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxpj9\" (UniqueName: \"kubernetes.io/projected/5f835847-daae-471e-a710-84ee468eff27-kube-api-access-fxpj9\") pod \"maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p\" (UID: \"5f835847-daae-471e-a710-84ee468eff27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:43.631064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.630944    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f835847-daae-471e-a710-84ee468eff27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p\" (UID: \"5f835847-daae-471e-a710-84ee468eff27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:43.631064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.631036    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f835847-daae-471e-a710-84ee468eff27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p\" (UID: \"5f835847-daae-471e-a710-84ee468eff27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:43.631657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.631626    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f835847-daae-471e-a710-84ee468eff27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p\" (UID: \"5f835847-daae-471e-a710-84ee468eff27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:43.633504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.633480    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f835847-daae-471e-a710-84ee468eff27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p\" (UID: \"5f835847-daae-471e-a710-84ee468eff27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:43.652925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.652897    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxpj9\" (UniqueName: \"kubernetes.io/projected/5f835847-daae-471e-a710-84ee468eff27-kube-api-access-fxpj9\") pod \"maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p\" (UID: \"5f835847-daae-471e-a710-84ee468eff27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:43.826911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:43.826840    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:44.169268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:44.169197    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:44.347485 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:44:44.347454    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f835847_daae_471e_a710_84ee468eff27.slice/crio-3db47a11a4257d37d3d59bc9e5899304cebec677c0ff913bc15b218d862df223 WatchSource:0}: Error finding container 3db47a11a4257d37d3d59bc9e5899304cebec677c0ff913bc15b218d862df223: Status 404 returned error can't find the container with id 3db47a11a4257d37d3d59bc9e5899304cebec677c0ff913bc15b218d862df223
Oct 29 22:44:44.352167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:44.352138    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"]
Oct 29 22:44:44.698386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:44.698354    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p" event={"ID":"5f835847-daae-471e-a710-84ee468eff27","Type":"ContainerStarted","Data":"3db47a11a4257d37d3d59bc9e5899304cebec677c0ff913bc15b218d862df223"}
Oct 29 22:44:45.795347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:45.790113    4860 generic.go:356] "Generic (PLEG): container finished" podID="5f835847-daae-471e-a710-84ee468eff27" containerID="09c59636a48887c35f4bc73aa4c26cd12a8f1c3a20970c4318a7b928b63b6b99" exitCode=0
Oct 29 22:44:45.795347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:45.790185    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p" event={"ID":"5f835847-daae-471e-a710-84ee468eff27","Type":"ContainerDied","Data":"09c59636a48887c35f4bc73aa4c26cd12a8f1c3a20970c4318a7b928b63b6b99"}
Oct 29 22:44:46.152630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:46.152549    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:46.371290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:46.371258    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:46.984952 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:46.984912    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:47.071002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.070924    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:47.124126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.124091    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f835847-daae-471e-a710-84ee468eff27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f835847-daae-471e-a710-84ee468eff27-maas-api-xks-ca\") pod \"5f835847-daae-471e-a710-84ee468eff27\" (UID: \"5f835847-daae-471e-a710-84ee468eff27\") "
Oct 29 22:44:47.124303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.124194    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f835847-daae-471e-a710-84ee468eff27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f835847-daae-471e-a710-84ee468eff27-maas-api-openshift-ca\") pod \"5f835847-daae-471e-a710-84ee468eff27\" (UID: \"5f835847-daae-471e-a710-84ee468eff27\") "
Oct 29 22:44:47.124501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.124478    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f835847-daae-471e-a710-84ee468eff27-kube-api-access-fxpj9\" (UniqueName: \"kubernetes.io/projected/5f835847-daae-471e-a710-84ee468eff27-kube-api-access-fxpj9\") pod \"5f835847-daae-471e-a710-84ee468eff27\" (UID: \"5f835847-daae-471e-a710-84ee468eff27\") "
Oct 29 22:44:47.125648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.125614    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f835847-daae-471e-a710-84ee468eff27-maas-api-openshift-ca" pod "5f835847-daae-471e-a710-84ee468eff27" (UID: "5f835847-daae-471e-a710-84ee468eff27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:44:47.150787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.150751    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f835847-daae-471e-a710-84ee468eff27-maas-api-xks-ca" pod "5f835847-daae-471e-a710-84ee468eff27" (UID: "5f835847-daae-471e-a710-84ee468eff27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:44:47.150924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.150790    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f835847-daae-471e-a710-84ee468eff27-kube-api-access-fxpj9" pod "5f835847-daae-471e-a710-84ee468eff27" (UID: "5f835847-daae-471e-a710-84ee468eff27"). InnerVolumeSpecName "kube-api-access-fxpj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:47.227442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.227406    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxpj9\" (UniqueName: \"kubernetes.io/projected/5f835847-daae-471e-a710-84ee468eff27-kube-api-access-fxpj9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:44:47.227442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.227440    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f835847-daae-471e-a710-84ee468eff27-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:44:47.227442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.227450    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f835847-daae-471e-a710-84ee468eff27-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:44:47.813180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.813146    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p" event={"ID":"5f835847-daae-471e-a710-84ee468eff27","Type":"ContainerDied","Data":"3db47a11a4257d37d3d59bc9e5899304cebec677c0ff913bc15b218d862df223"}
Oct 29 22:44:47.813383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.813185    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"
Oct 29 22:44:47.813383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:47.813204    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3db47a11a4257d37d3d59bc9e5899304cebec677c0ff913bc15b218d862df223"
Oct 29 22:44:48.058565 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:44:48.058529    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:44:48.379905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:48.379861    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:50.307998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:50.307964    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:51.165636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:51.165603    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:52.384118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:52.384086    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:53.410867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:53.410830    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:54.483614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:54.483578    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:54.658437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:54.658314    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:55.527201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:55.527168    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:56.770078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:56.770045    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:58.441024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:58.440997    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:58.777009 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:44:58.776834    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:00.214134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.214055    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"]
Oct 29 22:45:00.215011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.214993    4860 cpu_manager.go:414] "containerMap: removing container" podUID="5f835847-daae-471e-a710-84ee468eff27" containerName="revoke-keys"
Oct 29 22:45:00.215011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.215012    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f835847-daae-471e-a710-84ee468eff27" containerName="revoke-keys"
Oct 29 22:45:00.215155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.215146    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f835847-daae-471e-a710-84ee468eff27" containerName="revoke-keys"
Oct 29 22:45:00.219559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.219538    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"
Oct 29 22:45:00.222849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.222830    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"]
Oct 29 22:45:00.228589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.228228    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"
Oct 29 22:45:00.229449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.229430    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"]
Oct 29 22:45:00.234395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.234374    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"
Oct 29 22:45:00.235511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.235489    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"]
Oct 29 22:45:00.240099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.240079    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"
Oct 29 22:45:00.257316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.257296    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:45:00.272421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.272378    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"]
Oct 29 22:45:00.281774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.280691    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"]
Oct 29 22:45:00.284618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.284569    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"
Oct 29 22:45:00.286522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.286504    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"]
Oct 29 22:45:00.286655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.286634    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"
Oct 29 22:45:00.294521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.293420    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"]
Oct 29 22:45:00.294521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.293514    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"
Oct 29 22:45:00.300052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.300033    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"]
Oct 29 22:45:00.307877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.307861    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"]
Oct 29 22:45:00.318642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.315781    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"]
Oct 29 22:45:00.326789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.326766    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"]
Oct 29 22:45:00.335494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.335467    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgc6\" (UniqueName: \"kubernetes.io/projected/793ee6ac-d14d-480f-93d2-32bba5a4aed5-kube-api-access-7jgc6\") pod \"maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng\" (UID: \"793ee6ac-d14d-480f-93d2-32bba5a4aed5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"
Oct 29 22:45:00.335936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.335719    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f58gp\" (UniqueName: \"kubernetes.io/projected/c2580b08-b5bb-4306-a829-db8b7d832781-kube-api-access-f58gp\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4\" (UID: \"c2580b08-b5bb-4306-a829-db8b7d832781\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"
Oct 29 22:45:00.336228 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.336206    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f97838e-68b7-4252-93dd-f535f4cca7f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w\" (UID: \"2f97838e-68b7-4252-93dd-f535f4cca7f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"
Oct 29 22:45:00.336366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.336265    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/793ee6ac-d14d-480f-93d2-32bba5a4aed5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng\" (UID: \"793ee6ac-d14d-480f-93d2-32bba5a4aed5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"
Oct 29 22:45:00.336366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.336311    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlxz\" (UniqueName: \"kubernetes.io/projected/f00c15ba-e735-40a2-a1be-f32f20d8d7be-kube-api-access-qmlxz\") pod \"maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc\" (UID: \"f00c15ba-e735-40a2-a1be-f32f20d8d7be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"
Oct 29 22:45:00.336499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.336436    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dswm\" (UniqueName: \"kubernetes.io/projected/2f97838e-68b7-4252-93dd-f535f4cca7f1-kube-api-access-2dswm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w\" (UID: \"2f97838e-68b7-4252-93dd-f535f4cca7f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"
Oct 29 22:45:00.336499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.336483    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00c15ba-e735-40a2-a1be-f32f20d8d7be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc\" (UID: \"f00c15ba-e735-40a2-a1be-f32f20d8d7be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"
Oct 29 22:45:00.336616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.336597    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2580b08-b5bb-4306-a829-db8b7d832781-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4\" (UID: \"c2580b08-b5bb-4306-a829-db8b7d832781\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"
Oct 29 22:45:00.344565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.344480    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"]
Oct 29 22:45:00.351692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.351669    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"]
Oct 29 22:45:00.367517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.367494    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"]
Oct 29 22:45:00.373443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.373423    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"
Oct 29 22:45:00.412680 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.410787    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"]
Oct 29 22:45:00.438164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438129    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a54691-ca33-41d5-bdf2-f3d603727ee3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns\" (UID: \"41a54691-ca33-41d5-bdf2-f3d603727ee3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"
Oct 29 22:45:00.438363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438341    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f97838e-68b7-4252-93dd-f535f4cca7f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w\" (UID: \"2f97838e-68b7-4252-93dd-f535f4cca7f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"
Oct 29 22:45:00.438433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438397    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/793ee6ac-d14d-480f-93d2-32bba5a4aed5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng\" (UID: \"793ee6ac-d14d-480f-93d2-32bba5a4aed5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"
Oct 29 22:45:00.438494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438457    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlxz\" (UniqueName: \"kubernetes.io/projected/f00c15ba-e735-40a2-a1be-f32f20d8d7be-kube-api-access-qmlxz\") pod \"maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc\" (UID: \"f00c15ba-e735-40a2-a1be-f32f20d8d7be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"
Oct 29 22:45:00.438547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438495    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z8wl\" (UniqueName: \"kubernetes.io/projected/696b181a-c751-4741-8f95-86a613ef965c-kube-api-access-7z8wl\") pod \"maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594\" (UID: \"696b181a-c751-4741-8f95-86a613ef965c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"
Oct 29 22:45:00.438602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438574    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dswm\" (UniqueName: \"kubernetes.io/projected/2f97838e-68b7-4252-93dd-f535f4cca7f1-kube-api-access-2dswm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w\" (UID: \"2f97838e-68b7-4252-93dd-f535f4cca7f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"
Oct 29 22:45:00.438664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438615    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00c15ba-e735-40a2-a1be-f32f20d8d7be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc\" (UID: \"f00c15ba-e735-40a2-a1be-f32f20d8d7be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"
Oct 29 22:45:00.438730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438674    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2b9h\" (UniqueName: \"kubernetes.io/projected/8632936a-6257-4b14-9607-d7b18dfb85f7-kube-api-access-m2b9h\") pod \"maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk\" (UID: \"8632936a-6257-4b14-9607-d7b18dfb85f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"
Oct 29 22:45:00.438730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438698    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp6n7\" (UniqueName: \"kubernetes.io/projected/41a54691-ca33-41d5-bdf2-f3d603727ee3-kube-api-access-gp6n7\") pod \"maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns\" (UID: \"41a54691-ca33-41d5-bdf2-f3d603727ee3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"
Oct 29 22:45:00.438845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438751    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2580b08-b5bb-4306-a829-db8b7d832781-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4\" (UID: \"c2580b08-b5bb-4306-a829-db8b7d832781\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"
Oct 29 22:45:00.438845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438781    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8632936a-6257-4b14-9607-d7b18dfb85f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk\" (UID: \"8632936a-6257-4b14-9607-d7b18dfb85f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"
Oct 29 22:45:00.438979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438923    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696b181a-c751-4741-8f95-86a613ef965c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594\" (UID: \"696b181a-c751-4741-8f95-86a613ef965c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"
Oct 29 22:45:00.438979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.438968    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgc6\" (UniqueName: \"kubernetes.io/projected/793ee6ac-d14d-480f-93d2-32bba5a4aed5-kube-api-access-7jgc6\") pod \"maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng\" (UID: \"793ee6ac-d14d-480f-93d2-32bba5a4aed5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"
Oct 29 22:45:00.439125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.439017    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58gp\" (UniqueName: \"kubernetes.io/projected/c2580b08-b5bb-4306-a829-db8b7d832781-kube-api-access-f58gp\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4\" (UID: \"c2580b08-b5bb-4306-a829-db8b7d832781\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"
Oct 29 22:45:00.439648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.439620    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f97838e-68b7-4252-93dd-f535f4cca7f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w\" (UID: \"2f97838e-68b7-4252-93dd-f535f4cca7f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"
Oct 29 22:45:00.439760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.439706    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/793ee6ac-d14d-480f-93d2-32bba5a4aed5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng\" (UID: \"793ee6ac-d14d-480f-93d2-32bba5a4aed5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"
Oct 29 22:45:00.440238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.440216    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00c15ba-e735-40a2-a1be-f32f20d8d7be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc\" (UID: \"f00c15ba-e735-40a2-a1be-f32f20d8d7be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"
Oct 29 22:45:00.440344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.440307    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2580b08-b5bb-4306-a829-db8b7d832781-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4\" (UID: \"c2580b08-b5bb-4306-a829-db8b7d832781\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"
Oct 29 22:45:00.443217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.443192    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"]
Oct 29 22:45:00.450902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.450881    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"
Oct 29 22:45:00.465354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.465118    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"]
Oct 29 22:45:00.471513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.471491    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"
Oct 29 22:45:00.471662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.471640    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"]
Oct 29 22:45:00.486313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.486181    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58gp\" (UniqueName: \"kubernetes.io/projected/c2580b08-b5bb-4306-a829-db8b7d832781-kube-api-access-f58gp\") pod \"maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4\" (UID: \"c2580b08-b5bb-4306-a829-db8b7d832781\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"
Oct 29 22:45:00.491457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.487481    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgc6\" (UniqueName: \"kubernetes.io/projected/793ee6ac-d14d-480f-93d2-32bba5a4aed5-kube-api-access-7jgc6\") pod \"maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng\" (UID: \"793ee6ac-d14d-480f-93d2-32bba5a4aed5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"
Oct 29 22:45:00.493094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.493066    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlxz\" (UniqueName: \"kubernetes.io/projected/f00c15ba-e735-40a2-a1be-f32f20d8d7be-kube-api-access-qmlxz\") pod \"maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc\" (UID: \"f00c15ba-e735-40a2-a1be-f32f20d8d7be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"
Oct 29 22:45:00.493293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.493271    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"]
Oct 29 22:45:00.495676 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.495647    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dswm\" (UniqueName: \"kubernetes.io/projected/2f97838e-68b7-4252-93dd-f535f4cca7f1-kube-api-access-2dswm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w\" (UID: \"2f97838e-68b7-4252-93dd-f535f4cca7f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"
Oct 29 22:45:00.499600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.499584    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"
Oct 29 22:45:00.521335 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.519827    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"]
Oct 29 22:45:00.532347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.531473    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"]
Oct 29 22:45:00.536958 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.536935    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"
Oct 29 22:45:00.537300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.537267    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"
Oct 29 22:45:00.540397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.540372    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z8wl\" (UniqueName: \"kubernetes.io/projected/696b181a-c751-4741-8f95-86a613ef965c-kube-api-access-7z8wl\") pod \"maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594\" (UID: \"696b181a-c751-4741-8f95-86a613ef965c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"
Oct 29 22:45:00.540475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.540436    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s825d\" (UniqueName: \"kubernetes.io/projected/b032a2d6-c0e0-46cf-9265-e214157142f2-kube-api-access-s825d\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds\" (UID: \"b032a2d6-c0e0-46cf-9265-e214157142f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"
Oct 29 22:45:00.540544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.540532    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b032a2d6-c0e0-46cf-9265-e214157142f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds\" (UID: \"b032a2d6-c0e0-46cf-9265-e214157142f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"
Oct 29 22:45:00.540606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.540571    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2b9h\" (UniqueName: \"kubernetes.io/projected/8632936a-6257-4b14-9607-d7b18dfb85f7-kube-api-access-m2b9h\") pod \"maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk\" (UID: \"8632936a-6257-4b14-9607-d7b18dfb85f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"
Oct 29 22:45:00.540606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.540598    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp6n7\" (UniqueName: \"kubernetes.io/projected/41a54691-ca33-41d5-bdf2-f3d603727ee3-kube-api-access-gp6n7\") pod \"maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns\" (UID: \"41a54691-ca33-41d5-bdf2-f3d603727ee3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"
Oct 29 22:45:00.540715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.540658    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8632936a-6257-4b14-9607-d7b18dfb85f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk\" (UID: \"8632936a-6257-4b14-9607-d7b18dfb85f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"
Oct 29 22:45:00.541104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.540933    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9gfr\" (UniqueName: \"kubernetes.io/projected/b6184068-b2bf-487b-9b92-200dd0b4fd10-kube-api-access-r9gfr\") pod \"maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78\" (UID: \"b6184068-b2bf-487b-9b92-200dd0b4fd10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"
Oct 29 22:45:00.541104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.540988    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rzc\" (UniqueName: \"kubernetes.io/projected/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-kube-api-access-27rzc\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw\" (UID: \"455b3b28-47ae-40e5-a94d-c9ca25dc15c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"
Oct 29 22:45:00.541104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.541035    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696b181a-c751-4741-8f95-86a613ef965c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594\" (UID: \"696b181a-c751-4741-8f95-86a613ef965c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"
Oct 29 22:45:00.541316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.541139    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a54691-ca33-41d5-bdf2-f3d603727ee3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns\" (UID: \"41a54691-ca33-41d5-bdf2-f3d603727ee3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"
Oct 29 22:45:00.541316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.541234    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw\" (UID: \"455b3b28-47ae-40e5-a94d-c9ca25dc15c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"
Oct 29 22:45:00.541449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.541306    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6184068-b2bf-487b-9b92-200dd0b4fd10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78\" (UID: \"b6184068-b2bf-487b-9b92-200dd0b4fd10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"
Oct 29 22:45:00.541449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.541440    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8632936a-6257-4b14-9607-d7b18dfb85f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk\" (UID: \"8632936a-6257-4b14-9607-d7b18dfb85f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"
Oct 29 22:45:00.542160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.542139    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696b181a-c751-4741-8f95-86a613ef965c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594\" (UID: \"696b181a-c751-4741-8f95-86a613ef965c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"
Oct 29 22:45:00.542598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.542457    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a54691-ca33-41d5-bdf2-f3d603727ee3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns\" (UID: \"41a54691-ca33-41d5-bdf2-f3d603727ee3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"
Oct 29 22:45:00.545715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.545685    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"]
Oct 29 22:45:00.551197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.551170    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"
Oct 29 22:45:00.560351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.554397    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"
Oct 29 22:45:00.560351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.556288    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"
Oct 29 22:45:00.560351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.556441    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"]
Oct 29 22:45:00.564018 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.563992    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"
Oct 29 22:45:00.569249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.565794    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"]
Oct 29 22:45:00.583069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.579416    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"
Oct 29 22:45:00.588529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.587012    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp6n7\" (UniqueName: \"kubernetes.io/projected/41a54691-ca33-41d5-bdf2-f3d603727ee3-kube-api-access-gp6n7\") pod \"maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns\" (UID: \"41a54691-ca33-41d5-bdf2-f3d603727ee3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"
Oct 29 22:45:00.595575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.595177    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"]
Oct 29 22:45:00.606425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.602115    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z8wl\" (UniqueName: \"kubernetes.io/projected/696b181a-c751-4741-8f95-86a613ef965c-kube-api-access-7z8wl\") pod \"maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594\" (UID: \"696b181a-c751-4741-8f95-86a613ef965c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"
Oct 29 22:45:00.609905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.607230    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"]
Oct 29 22:45:00.613931 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.613685    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"
Oct 29 22:45:00.617768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.617741    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"]
Oct 29 22:45:00.617876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.617828    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"
Oct 29 22:45:00.626988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.626951    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"
Oct 29 22:45:00.636099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.635051    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"]
Oct 29 22:45:00.639662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.639639    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2b9h\" (UniqueName: \"kubernetes.io/projected/8632936a-6257-4b14-9607-d7b18dfb85f7-kube-api-access-m2b9h\") pod \"maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk\" (UID: \"8632936a-6257-4b14-9607-d7b18dfb85f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.645158    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw\" (UID: \"455b3b28-47ae-40e5-a94d-c9ca25dc15c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.645209    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6184068-b2bf-487b-9b92-200dd0b4fd10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78\" (UID: \"b6184068-b2bf-487b-9b92-200dd0b4fd10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.645273    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brrfk\" (UniqueName: \"kubernetes.io/projected/3b8a9038-c7c7-497b-b177-8c8342af29a4-kube-api-access-brrfk\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2\" (UID: \"3b8a9038-c7c7-497b-b177-8c8342af29a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.645408    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s825d\" (UniqueName: \"kubernetes.io/projected/b032a2d6-c0e0-46cf-9265-e214157142f2-kube-api-access-s825d\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds\" (UID: \"b032a2d6-c0e0-46cf-9265-e214157142f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.645443    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8a9038-c7c7-497b-b177-8c8342af29a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2\" (UID: \"3b8a9038-c7c7-497b-b177-8c8342af29a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.645473    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46509b71-e375-40ca-9a04-d35ddaa6e8bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs\" (UID: \"46509b71-e375-40ca-9a04-d35ddaa6e8bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.645512    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dhbq\" (UniqueName: \"kubernetes.io/projected/46509b71-e375-40ca-9a04-d35ddaa6e8bc-kube-api-access-5dhbq\") pod \"maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs\" (UID: \"46509b71-e375-40ca-9a04-d35ddaa6e8bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.645567    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxbzr\" (UniqueName: \"kubernetes.io/projected/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-kube-api-access-gxbzr\") pod \"maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5\" (UID: \"80f7cc19-3ccf-4e4a-8d30-8f76251885b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.645610    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b032a2d6-c0e0-46cf-9265-e214157142f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds\" (UID: \"b032a2d6-c0e0-46cf-9265-e214157142f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.645745    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5\" (UID: \"80f7cc19-3ccf-4e4a-8d30-8f76251885b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.645775    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9gfr\" (UniqueName: \"kubernetes.io/projected/b6184068-b2bf-487b-9b92-200dd0b4fd10-kube-api-access-r9gfr\") pod \"maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78\" (UID: \"b6184068-b2bf-487b-9b92-200dd0b4fd10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.645821    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rzc\" (UniqueName: \"kubernetes.io/projected/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-kube-api-access-27rzc\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw\" (UID: \"455b3b28-47ae-40e5-a94d-c9ca25dc15c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.646123    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"]
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.646996    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw\" (UID: \"455b3b28-47ae-40e5-a94d-c9ca25dc15c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.647838    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6184068-b2bf-487b-9b92-200dd0b4fd10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78\" (UID: \"b6184068-b2bf-487b-9b92-200dd0b4fd10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.648139    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"
Oct 29 22:45:00.653786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.650697    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b032a2d6-c0e0-46cf-9265-e214157142f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds\" (UID: \"b032a2d6-c0e0-46cf-9265-e214157142f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"
Oct 29 22:45:00.697432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.697302    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s825d\" (UniqueName: \"kubernetes.io/projected/b032a2d6-c0e0-46cf-9265-e214157142f2-kube-api-access-s825d\") pod \"maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds\" (UID: \"b032a2d6-c0e0-46cf-9265-e214157142f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"
Oct 29 22:45:00.703747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.703718    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9gfr\" (UniqueName: \"kubernetes.io/projected/b6184068-b2bf-487b-9b92-200dd0b4fd10-kube-api-access-r9gfr\") pod \"maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78\" (UID: \"b6184068-b2bf-487b-9b92-200dd0b4fd10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"
Oct 29 22:45:00.706450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.704655    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rzc\" (UniqueName: \"kubernetes.io/projected/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-kube-api-access-27rzc\") pod \"maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw\" (UID: \"455b3b28-47ae-40e5-a94d-c9ca25dc15c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"
Oct 29 22:45:00.778689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.777762    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"
Oct 29 22:45:00.794566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.794364    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.795628    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46509b71-e375-40ca-9a04-d35ddaa6e8bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs\" (UID: \"46509b71-e375-40ca-9a04-d35ddaa6e8bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.795686    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dhbq\" (UniqueName: \"kubernetes.io/projected/46509b71-e375-40ca-9a04-d35ddaa6e8bc-kube-api-access-5dhbq\") pod \"maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs\" (UID: \"46509b71-e375-40ca-9a04-d35ddaa6e8bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.795727    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767\" (UID: \"6d1c0aa4-106c-44cf-8c66-c94717ba5c48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.795800    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxbzr\" (UniqueName: \"kubernetes.io/projected/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-kube-api-access-gxbzr\") pod \"maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5\" (UID: \"80f7cc19-3ccf-4e4a-8d30-8f76251885b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.795931    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-647wd\" (UniqueName: \"kubernetes.io/projected/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-kube-api-access-647wd\") pod \"maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj\" (UID: \"1276947c-a4b8-4eb3-8aaf-2d589e81b84c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.795994    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5\" (UID: \"80f7cc19-3ccf-4e4a-8d30-8f76251885b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.796183    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj\" (UID: \"1276947c-a4b8-4eb3-8aaf-2d589e81b84c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.796239    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2spwl\" (UniqueName: \"kubernetes.io/projected/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-kube-api-access-2spwl\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767\" (UID: \"6d1c0aa4-106c-44cf-8c66-c94717ba5c48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.796270    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brrfk\" (UniqueName: \"kubernetes.io/projected/3b8a9038-c7c7-497b-b177-8c8342af29a4-kube-api-access-brrfk\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2\" (UID: \"3b8a9038-c7c7-497b-b177-8c8342af29a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.796475    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8a9038-c7c7-497b-b177-8c8342af29a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2\" (UID: \"3b8a9038-c7c7-497b-b177-8c8342af29a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.797239    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8a9038-c7c7-497b-b177-8c8342af29a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2\" (UID: \"3b8a9038-c7c7-497b-b177-8c8342af29a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.797843    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46509b71-e375-40ca-9a04-d35ddaa6e8bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs\" (UID: \"46509b71-e375-40ca-9a04-d35ddaa6e8bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"
Oct 29 22:45:00.799815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.799141    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5\" (UID: \"80f7cc19-3ccf-4e4a-8d30-8f76251885b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"
Oct 29 22:45:00.817386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.817075    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:00.864548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.864486    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dhbq\" (UniqueName: \"kubernetes.io/projected/46509b71-e375-40ca-9a04-d35ddaa6e8bc-kube-api-access-5dhbq\") pod \"maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs\" (UID: \"46509b71-e375-40ca-9a04-d35ddaa6e8bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"
Oct 29 22:45:00.865466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.865418    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxbzr\" (UniqueName: \"kubernetes.io/projected/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-kube-api-access-gxbzr\") pod \"maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5\" (UID: \"80f7cc19-3ccf-4e4a-8d30-8f76251885b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"
Oct 29 22:45:00.884998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.881004    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brrfk\" (UniqueName: \"kubernetes.io/projected/3b8a9038-c7c7-497b-b177-8c8342af29a4-kube-api-access-brrfk\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2\" (UID: \"3b8a9038-c7c7-497b-b177-8c8342af29a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"
Oct 29 22:45:00.898922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.898787    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767\" (UID: \"6d1c0aa4-106c-44cf-8c66-c94717ba5c48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"
Oct 29 22:45:00.899576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.899181    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-647wd\" (UniqueName: \"kubernetes.io/projected/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-kube-api-access-647wd\") pod \"maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj\" (UID: \"1276947c-a4b8-4eb3-8aaf-2d589e81b84c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"
Oct 29 22:45:00.899576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.899406    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj\" (UID: \"1276947c-a4b8-4eb3-8aaf-2d589e81b84c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"
Oct 29 22:45:00.899576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.899462    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2spwl\" (UniqueName: \"kubernetes.io/projected/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-kube-api-access-2spwl\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767\" (UID: \"6d1c0aa4-106c-44cf-8c66-c94717ba5c48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"
Oct 29 22:45:00.900936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.900850    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767\" (UID: \"6d1c0aa4-106c-44cf-8c66-c94717ba5c48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"
Oct 29 22:45:00.901911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.901871    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj\" (UID: \"1276947c-a4b8-4eb3-8aaf-2d589e81b84c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"
Oct 29 22:45:00.931570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.931475    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2spwl\" (UniqueName: \"kubernetes.io/projected/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-kube-api-access-2spwl\") pod \"maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767\" (UID: \"6d1c0aa4-106c-44cf-8c66-c94717ba5c48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"
Oct 29 22:45:00.938234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.938205    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-647wd\" (UniqueName: \"kubernetes.io/projected/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-kube-api-access-647wd\") pod \"maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj\" (UID: \"1276947c-a4b8-4eb3-8aaf-2d589e81b84c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"
Oct 29 22:45:00.976378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.976241    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"
Oct 29 22:45:00.994401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:00.993541    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"
Oct 29 22:45:01.014583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.014552    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"
Oct 29 22:45:01.091556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.091363    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"
Oct 29 22:45:01.104846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.104813    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"
Oct 29 22:45:01.138348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.138187    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"]
Oct 29 22:45:01.160983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.158134    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"
Oct 29 22:45:01.167732 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:45:01.167300    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f97838e_68b7_4252_93dd_f535f4cca7f1.slice/crio-fdc3c469ea08ece84b1d795f6536a278e5404e501dba271d7ba3c2be5c665ade WatchSource:0}: Error finding container fdc3c469ea08ece84b1d795f6536a278e5404e501dba271d7ba3c2be5c665ade: Status 404 returned error can't find the container with id fdc3c469ea08ece84b1d795f6536a278e5404e501dba271d7ba3c2be5c665ade
Oct 29 22:45:01.367812 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:45:01.367773    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793ee6ac_d14d_480f_93d2_32bba5a4aed5.slice/crio-4720a73507db5ad8eb1037de1a090087677d68d2e1bfa1aff7cf38fbbf34d1d6 WatchSource:0}: Error finding container 4720a73507db5ad8eb1037de1a090087677d68d2e1bfa1aff7cf38fbbf34d1d6: Status 404 returned error can't find the container with id 4720a73507db5ad8eb1037de1a090087677d68d2e1bfa1aff7cf38fbbf34d1d6
Oct 29 22:45:01.374192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.374165    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"]
Oct 29 22:45:01.524190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.524157    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"]
Oct 29 22:45:01.578241 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:45:01.578171    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00c15ba_e735_40a2_a1be_f32f20d8d7be.slice/crio-19167d5487b9e03d501b1517619c1dc3f0818f81e1312a93f1952914c07ace24 WatchSource:0}: Error finding container 19167d5487b9e03d501b1517619c1dc3f0818f81e1312a93f1952914c07ace24: Status 404 returned error can't find the container with id 19167d5487b9e03d501b1517619c1dc3f0818f81e1312a93f1952914c07ace24
Oct 29 22:45:01.583196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.582102    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"]
Oct 29 22:45:01.640722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.640690    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"]
Oct 29 22:45:01.641740 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:45:01.641712    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696b181a_c751_4741_8f95_86a613ef965c.slice/crio-de758f9995fff8450fb0dbfd2273869908f6b6da5807a8904d64611c129a6b7b WatchSource:0}: Error finding container de758f9995fff8450fb0dbfd2273869908f6b6da5807a8904d64611c129a6b7b: Status 404 returned error can't find the container with id de758f9995fff8450fb0dbfd2273869908f6b6da5807a8904d64611c129a6b7b
Oct 29 22:45:01.663899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.663870    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"]
Oct 29 22:45:01.684479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.684450    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"]
Oct 29 22:45:01.696694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.692520    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"]
Oct 29 22:45:01.696694 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:45:01.692913    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8632936a_6257_4b14_9607_d7b18dfb85f7.slice/crio-86b2d8855ac0263075c4d3be849dc46a6be2d818434d7703fac1a75c10e0a10e WatchSource:0}: Error finding container 86b2d8855ac0263075c4d3be849dc46a6be2d818434d7703fac1a75c10e0a10e: Status 404 returned error can't find the container with id 86b2d8855ac0263075c4d3be849dc46a6be2d818434d7703fac1a75c10e0a10e
Oct 29 22:45:01.700980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.700957    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"]
Oct 29 22:45:01.730160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.730133    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"]
Oct 29 22:45:01.804679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.804646    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"]
Oct 29 22:45:01.804958 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:45:01.804933    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1c0aa4_106c_44cf_8c66_c94717ba5c48.slice/crio-3cda374cf88852b3ec75f40a7b0b1fa0c4597cf8e381c955c6468e1037258a6d WatchSource:0}: Error finding container 3cda374cf88852b3ec75f40a7b0b1fa0c4597cf8e381c955c6468e1037258a6d: Status 404 returned error can't find the container with id 3cda374cf88852b3ec75f40a7b0b1fa0c4597cf8e381c955c6468e1037258a6d
Oct 29 22:45:01.810929 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:45:01.810880    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8a9038_c7c7_497b_b177_8c8342af29a4.slice/crio-b01c9296a6f4e23d8664ea886a2999fd7bcca7f014893b6f50df76e3a5869e07 WatchSource:0}: Error finding container b01c9296a6f4e23d8664ea886a2999fd7bcca7f014893b6f50df76e3a5869e07: Status 404 returned error can't find the container with id b01c9296a6f4e23d8664ea886a2999fd7bcca7f014893b6f50df76e3a5869e07
Oct 29 22:45:01.812868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.812843    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"]
Oct 29 22:45:01.837676 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.837647    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"]
Oct 29 22:45:01.838245 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:45:01.838215    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46509b71_e375_40ca_9a04_d35ddaa6e8bc.slice/crio-a2c154991fe8cea6776ddd8c18f1002ed71c63dc55f3c07eb6f242cdb222121e WatchSource:0}: Error finding container a2c154991fe8cea6776ddd8c18f1002ed71c63dc55f3c07eb6f242cdb222121e: Status 404 returned error can't find the container with id a2c154991fe8cea6776ddd8c18f1002ed71c63dc55f3c07eb6f242cdb222121e
Oct 29 22:45:01.860276 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:45:01.860245    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f7cc19_3ccf_4e4a_8d30_8f76251885b0.slice/crio-50c2b140247fc99da067615a10cd296a63ea5a13a135e584e3ddf4208a3620dc WatchSource:0}: Error finding container 50c2b140247fc99da067615a10cd296a63ea5a13a135e584e3ddf4208a3620dc: Status 404 returned error can't find the container with id 50c2b140247fc99da067615a10cd296a63ea5a13a135e584e3ddf4208a3620dc
Oct 29 22:45:01.860866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.860829    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"]
Oct 29 22:45:01.894714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:01.894673    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"]
Oct 29 22:45:02.041794 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.041759    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds" event={"ID":"b032a2d6-c0e0-46cf-9265-e214157142f2","Type":"ContainerStarted","Data":"20e5058cee851781ad8e851c6cf90a95ef6e5f4b2b19e8c6cbfe01da88cae8e2"}
Oct 29 22:45:02.043266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.043240    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc" event={"ID":"f00c15ba-e735-40a2-a1be-f32f20d8d7be","Type":"ContainerStarted","Data":"19167d5487b9e03d501b1517619c1dc3f0818f81e1312a93f1952914c07ace24"}
Oct 29 22:45:02.044580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.044560    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5" event={"ID":"80f7cc19-3ccf-4e4a-8d30-8f76251885b0","Type":"ContainerStarted","Data":"50c2b140247fc99da067615a10cd296a63ea5a13a135e584e3ddf4208a3620dc"}
Oct 29 22:45:02.046103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.046060    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns" event={"ID":"41a54691-ca33-41d5-bdf2-f3d603727ee3","Type":"ContainerStarted","Data":"a6c0513bcc6fcb9c9488c801b727efe559c84faf611813954ceb192b09c6a72d"}
Oct 29 22:45:02.047572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.047548    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk" event={"ID":"8632936a-6257-4b14-9607-d7b18dfb85f7","Type":"ContainerStarted","Data":"86b2d8855ac0263075c4d3be849dc46a6be2d818434d7703fac1a75c10e0a10e"}
Oct 29 22:45:02.049041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.049013    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs" event={"ID":"46509b71-e375-40ca-9a04-d35ddaa6e8bc","Type":"ContainerStarted","Data":"a2c154991fe8cea6776ddd8c18f1002ed71c63dc55f3c07eb6f242cdb222121e"}
Oct 29 22:45:02.050371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.050348    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2" event={"ID":"3b8a9038-c7c7-497b-b177-8c8342af29a4","Type":"ContainerStarted","Data":"b01c9296a6f4e23d8664ea886a2999fd7bcca7f014893b6f50df76e3a5869e07"}
Oct 29 22:45:02.051667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.051645    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594" event={"ID":"696b181a-c751-4741-8f95-86a613ef965c","Type":"ContainerStarted","Data":"de758f9995fff8450fb0dbfd2273869908f6b6da5807a8904d64611c129a6b7b"}
Oct 29 22:45:02.053171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.053148    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" event={"ID":"b6184068-b2bf-487b-9b92-200dd0b4fd10","Type":"ContainerStarted","Data":"3eaf85017f2bec319a8a9295f1598acd4afa5292198a1fccae203bc65cf5d0fe"}
Oct 29 22:45:02.054674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.054634    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4" event={"ID":"c2580b08-b5bb-4306-a829-db8b7d832781","Type":"ContainerStarted","Data":"92d1c99395e13b8d3f200dec32ecbed83e7151272cfb62b0b6d2d277a46d31a4"}
Oct 29 22:45:02.056109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.056089    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj" event={"ID":"1276947c-a4b8-4eb3-8aaf-2d589e81b84c","Type":"ContainerStarted","Data":"e7f829a0aa574154595a304d66b1a3324fb25ef7a12c21fb1df8bcc2d282611d"}
Oct 29 22:45:02.057514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.057489    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w" event={"ID":"2f97838e-68b7-4252-93dd-f535f4cca7f1","Type":"ContainerStarted","Data":"fdc3c469ea08ece84b1d795f6536a278e5404e501dba271d7ba3c2be5c665ade"}
Oct 29 22:45:02.058978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.058951    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767" event={"ID":"6d1c0aa4-106c-44cf-8c66-c94717ba5c48","Type":"ContainerStarted","Data":"3cda374cf88852b3ec75f40a7b0b1fa0c4597cf8e381c955c6468e1037258a6d"}
Oct 29 22:45:02.060726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.060706    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng" event={"ID":"793ee6ac-d14d-480f-93d2-32bba5a4aed5","Type":"ContainerStarted","Data":"8ac038aba14417f4b7ad721d15f5b1d4271862ace9e38f62d82e571ca79e776b"}
Oct 29 22:45:02.060808 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.060741    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng" event={"ID":"793ee6ac-d14d-480f-93d2-32bba5a4aed5","Type":"ContainerStarted","Data":"4720a73507db5ad8eb1037de1a090087677d68d2e1bfa1aff7cf38fbbf34d1d6"}
Oct 29 22:45:02.111088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.111043    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng" podStartSLOduration=2.111029787 podStartE2EDuration="2.111029787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.104298572 +0000 UTC m=+1844.129074704" watchObservedRunningTime="2026-10-29 22:45:02.111029787 +0000 UTC m=+1844.135805918"
Oct 29 22:45:02.513194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:02.513157    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:03.088776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.088734    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds" event={"ID":"b032a2d6-c0e0-46cf-9265-e214157142f2","Type":"ContainerStarted","Data":"d54d9935ff0db8e254472c894090d6df9fd898b8098cc8a6b364107f91026f0d"}
Oct 29 22:45:03.092055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.092012    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc" event={"ID":"f00c15ba-e735-40a2-a1be-f32f20d8d7be","Type":"ContainerStarted","Data":"15f3a6a88f8571bbca21646e9422009efe8ffb68d857f07e91fc4ff4121f3acd"}
Oct 29 22:45:03.094041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.094010    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5" event={"ID":"80f7cc19-3ccf-4e4a-8d30-8f76251885b0","Type":"ContainerStarted","Data":"b06b10da5de40b33bc3bd7d3b1c60e04a2438a7603656cdc2369b2d865f3b54b"}
Oct 29 22:45:03.095802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.095774    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns" event={"ID":"41a54691-ca33-41d5-bdf2-f3d603727ee3","Type":"ContainerStarted","Data":"fed0e151ebd088a92c5f686e467d7a31a01532fa627616180a6d756661e7fe44"}
Oct 29 22:45:03.097594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.097562    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk" event={"ID":"8632936a-6257-4b14-9607-d7b18dfb85f7","Type":"ContainerStarted","Data":"3f2d5ddd05d98b2e7007d1aac07d20432bf9231798fa84556f6d28d183b15ce8"}
Oct 29 22:45:03.099271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.099247    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs" event={"ID":"46509b71-e375-40ca-9a04-d35ddaa6e8bc","Type":"ContainerStarted","Data":"91eb9193cd7d73033533db96362ab02079ae9ebab54b82f8c504655626c0f6ae"}
Oct 29 22:45:03.100958 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.100932    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2" event={"ID":"3b8a9038-c7c7-497b-b177-8c8342af29a4","Type":"ContainerStarted","Data":"c43dcf1a42c87ea62ee46d9aa634df84bc234994fcf4579ffc8c96e3c14c4fda"}
Oct 29 22:45:03.102442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.102416    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594" event={"ID":"696b181a-c751-4741-8f95-86a613ef965c","Type":"ContainerStarted","Data":"240fa60e0b0dd8a12e4da16b1be47f0a918d73d082aa06196ddcaae422be67fb"}
Oct 29 22:45:03.104067 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.104040    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" event={"ID":"b6184068-b2bf-487b-9b92-200dd0b4fd10","Type":"ContainerStarted","Data":"3a7eff867e1393646252b0de103fa14953ba09cc55fb622b3f41ea5307cd15d2"}
Oct 29 22:45:03.105692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.105671    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw" event={"ID":"455b3b28-47ae-40e5-a94d-c9ca25dc15c0","Type":"ContainerStarted","Data":"28434ff056e4e2bd86b4053cfabbbd783175e4189c5f509b29d0273143c23432"}
Oct 29 22:45:03.105788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.105713    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw" event={"ID":"455b3b28-47ae-40e5-a94d-c9ca25dc15c0","Type":"ContainerStarted","Data":"fbfa8f4798f2ab32a80efac5246b366e7568c4bf7e5a63d5b4e1b0e4a63511bc"}
Oct 29 22:45:03.107369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.107343    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4" event={"ID":"c2580b08-b5bb-4306-a829-db8b7d832781","Type":"ContainerStarted","Data":"71572e5d961fea33a89fd8a9f1fd0b734b466e31870007e04e94a0157d60c6b3"}
Oct 29 22:45:03.108890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.108869    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj" event={"ID":"1276947c-a4b8-4eb3-8aaf-2d589e81b84c","Type":"ContainerStarted","Data":"7e967827e10b9479f29579a8e936ab0b153b71adaa397ea1243caa9a08bc129a"}
Oct 29 22:45:03.110581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.110559    4860 generic.go:356] "Generic (PLEG): container finished" podID="2f97838e-68b7-4252-93dd-f535f4cca7f1" containerID="3a9ff3348e45c46d682585ca988d0b027c39d07d500a00e9f9a9460c277e055a" exitCode=60
Oct 29 22:45:03.110690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.110647    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w" event={"ID":"2f97838e-68b7-4252-93dd-f535f4cca7f1","Type":"ContainerDied","Data":"3a9ff3348e45c46d682585ca988d0b027c39d07d500a00e9f9a9460c277e055a"}
Oct 29 22:45:03.110829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.110809    4860 scope.go:122] "RemoveContainer" containerID="3a9ff3348e45c46d682585ca988d0b027c39d07d500a00e9f9a9460c277e055a"
Oct 29 22:45:03.112441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.112411    4860 generic.go:356] "Generic (PLEG): container finished" podID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" containerID="2735260ee7131f54d786ecfbaa9b4b9e782287847fc10973e21cc881b62820aa" exitCode=60
Oct 29 22:45:03.112545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.112448    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767" event={"ID":"6d1c0aa4-106c-44cf-8c66-c94717ba5c48","Type":"ContainerDied","Data":"2735260ee7131f54d786ecfbaa9b4b9e782287847fc10973e21cc881b62820aa"}
Oct 29 22:45:03.112667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.112653    4860 scope.go:122] "RemoveContainer" containerID="2735260ee7131f54d786ecfbaa9b4b9e782287847fc10973e21cc881b62820aa"
Oct 29 22:45:03.114391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.114358    4860 generic.go:356] "Generic (PLEG): container finished" podID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" containerID="8ac038aba14417f4b7ad721d15f5b1d4271862ace9e38f62d82e571ca79e776b" exitCode=60
Oct 29 22:45:03.114487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.114426    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng" event={"ID":"793ee6ac-d14d-480f-93d2-32bba5a4aed5","Type":"ContainerDied","Data":"8ac038aba14417f4b7ad721d15f5b1d4271862ace9e38f62d82e571ca79e776b"}
Oct 29 22:45:03.114715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.114697    4860 scope.go:122] "RemoveContainer" containerID="8ac038aba14417f4b7ad721d15f5b1d4271862ace9e38f62d82e571ca79e776b"
Oct 29 22:45:03.135577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.135534    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds" podStartSLOduration=3.135518537 podStartE2EDuration="3.135518537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.132020997 +0000 UTC m=+1845.156797139" watchObservedRunningTime="2026-10-29 22:45:03.135518537 +0000 UTC m=+1845.160294669"
Oct 29 22:45:03.183400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.183334    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk" podStartSLOduration=3.183302723 podStartE2EDuration="3.183302723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.179732901 +0000 UTC m=+1845.204509032" watchObservedRunningTime="2026-10-29 22:45:03.183302723 +0000 UTC m=+1845.208078857"
Oct 29 22:45:03.224820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.224771    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs" podStartSLOduration=3.224758186 podStartE2EDuration="3.224758186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.221388041 +0000 UTC m=+1845.246164189" watchObservedRunningTime="2026-10-29 22:45:03.224758186 +0000 UTC m=+1845.249534317"
Oct 29 22:45:03.266314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.266229    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw" podStartSLOduration=3.266215976 podStartE2EDuration="3.266215976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.262953843 +0000 UTC m=+1845.287729974" watchObservedRunningTime="2026-10-29 22:45:03.266215976 +0000 UTC m=+1845.290992107"
Oct 29 22:45:03.309925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.309876    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc" podStartSLOduration=3.309863692 podStartE2EDuration="3.309863692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.30475391 +0000 UTC m=+1845.329530043" watchObservedRunningTime="2026-10-29 22:45:03.309863692 +0000 UTC m=+1845.334639850"
Oct 29 22:45:03.346941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.346892    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns" podStartSLOduration=3.346878188 podStartE2EDuration="3.346878188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.345908284 +0000 UTC m=+1845.370684417" watchObservedRunningTime="2026-10-29 22:45:03.346878188 +0000 UTC m=+1845.371654371"
Oct 29 22:45:03.388310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.388265    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5" podStartSLOduration=3.3882527590000002 podStartE2EDuration="3.388252759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.385754375 +0000 UTC m=+1845.410530526" watchObservedRunningTime="2026-10-29 22:45:03.388252759 +0000 UTC m=+1845.413028889"
Oct 29 22:45:03.429318 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.429269    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2" podStartSLOduration=3.429256217 podStartE2EDuration="3.429256217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.427619712 +0000 UTC m=+1845.452395844" watchObservedRunningTime="2026-10-29 22:45:03.429256217 +0000 UTC m=+1845.454032385"
Oct 29 22:45:03.470599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.470546    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj" podStartSLOduration=3.470533683 podStartE2EDuration="3.470533683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.467797179 +0000 UTC m=+1845.492573310" watchObservedRunningTime="2026-10-29 22:45:03.470533683 +0000 UTC m=+1845.495309813"
Oct 29 22:45:03.598477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.598425    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" podStartSLOduration=3.59840837 podStartE2EDuration="3.59840837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.594893152 +0000 UTC m=+1845.619669285" watchObservedRunningTime="2026-10-29 22:45:03.59840837 +0000 UTC m=+1845.623184505"
Oct 29 22:45:03.704065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.704021    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594" podStartSLOduration=3.704008087 podStartE2EDuration="3.704008087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.701941085 +0000 UTC m=+1845.726717236" watchObservedRunningTime="2026-10-29 22:45:03.704008087 +0000 UTC m=+1845.728784218"
Oct 29 22:45:03.753872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:03.753828    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4" podStartSLOduration=3.753814393 podStartE2EDuration="3.753814393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.749505482 +0000 UTC m=+1845.774281634" watchObservedRunningTime="2026-10-29 22:45:03.753814393 +0000 UTC m=+1845.778590524"
Oct 29 22:45:04.124730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.124698    4860 generic.go:356] "Generic (PLEG): container finished" podID="80f7cc19-3ccf-4e4a-8d30-8f76251885b0" containerID="b06b10da5de40b33bc3bd7d3b1c60e04a2438a7603656cdc2369b2d865f3b54b" exitCode=60
Oct 29 22:45:04.124900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.124769    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5" event={"ID":"80f7cc19-3ccf-4e4a-8d30-8f76251885b0","Type":"ContainerDied","Data":"b06b10da5de40b33bc3bd7d3b1c60e04a2438a7603656cdc2369b2d865f3b54b"}
Oct 29 22:45:04.124963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.124948    4860 scope.go:122] "RemoveContainer" containerID="b06b10da5de40b33bc3bd7d3b1c60e04a2438a7603656cdc2369b2d865f3b54b"
Oct 29 22:45:04.126571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.126542    4860 generic.go:356] "Generic (PLEG): container finished" podID="41a54691-ca33-41d5-bdf2-f3d603727ee3" containerID="fed0e151ebd088a92c5f686e467d7a31a01532fa627616180a6d756661e7fe44" exitCode=60
Oct 29 22:45:04.126571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.126572    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns" event={"ID":"41a54691-ca33-41d5-bdf2-f3d603727ee3","Type":"ContainerDied","Data":"fed0e151ebd088a92c5f686e467d7a31a01532fa627616180a6d756661e7fe44"}
Oct 29 22:45:04.126850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.126839    4860 scope.go:122] "RemoveContainer" containerID="fed0e151ebd088a92c5f686e467d7a31a01532fa627616180a6d756661e7fe44"
Oct 29 22:45:04.128464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.128434    4860 generic.go:356] "Generic (PLEG): container finished" podID="8632936a-6257-4b14-9607-d7b18dfb85f7" containerID="3f2d5ddd05d98b2e7007d1aac07d20432bf9231798fa84556f6d28d183b15ce8" exitCode=60
Oct 29 22:45:04.128547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.128525    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk" event={"ID":"8632936a-6257-4b14-9607-d7b18dfb85f7","Type":"ContainerDied","Data":"3f2d5ddd05d98b2e7007d1aac07d20432bf9231798fa84556f6d28d183b15ce8"}
Oct 29 22:45:04.128802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.128738    4860 scope.go:122] "RemoveContainer" containerID="3f2d5ddd05d98b2e7007d1aac07d20432bf9231798fa84556f6d28d183b15ce8"
Oct 29 22:45:04.130389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.130301    4860 generic.go:356] "Generic (PLEG): container finished" podID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" containerID="91eb9193cd7d73033533db96362ab02079ae9ebab54b82f8c504655626c0f6ae" exitCode=60
Oct 29 22:45:04.130389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.130365    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs" event={"ID":"46509b71-e375-40ca-9a04-d35ddaa6e8bc","Type":"ContainerDied","Data":"91eb9193cd7d73033533db96362ab02079ae9ebab54b82f8c504655626c0f6ae"}
Oct 29 22:45:04.130722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.130640    4860 scope.go:122] "RemoveContainer" containerID="91eb9193cd7d73033533db96362ab02079ae9ebab54b82f8c504655626c0f6ae"
Oct 29 22:45:04.132242 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.132220    4860 generic.go:356] "Generic (PLEG): container finished" podID="3b8a9038-c7c7-497b-b177-8c8342af29a4" containerID="c43dcf1a42c87ea62ee46d9aa634df84bc234994fcf4579ffc8c96e3c14c4fda" exitCode=60
Oct 29 22:45:04.132380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.132300    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2" event={"ID":"3b8a9038-c7c7-497b-b177-8c8342af29a4","Type":"ContainerDied","Data":"c43dcf1a42c87ea62ee46d9aa634df84bc234994fcf4579ffc8c96e3c14c4fda"}
Oct 29 22:45:04.132543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.132529    4860 scope.go:122] "RemoveContainer" containerID="c43dcf1a42c87ea62ee46d9aa634df84bc234994fcf4579ffc8c96e3c14c4fda"
Oct 29 22:45:04.134039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.134016    4860 generic.go:356] "Generic (PLEG): container finished" podID="696b181a-c751-4741-8f95-86a613ef965c" containerID="240fa60e0b0dd8a12e4da16b1be47f0a918d73d082aa06196ddcaae422be67fb" exitCode=60
Oct 29 22:45:04.134139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.134086    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594" event={"ID":"696b181a-c751-4741-8f95-86a613ef965c","Type":"ContainerDied","Data":"240fa60e0b0dd8a12e4da16b1be47f0a918d73d082aa06196ddcaae422be67fb"}
Oct 29 22:45:04.134313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.134297    4860 scope.go:122] "RemoveContainer" containerID="240fa60e0b0dd8a12e4da16b1be47f0a918d73d082aa06196ddcaae422be67fb"
Oct 29 22:45:04.135789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.135772    4860 generic.go:356] "Generic (PLEG): container finished" podID="b6184068-b2bf-487b-9b92-200dd0b4fd10" containerID="3a7eff867e1393646252b0de103fa14953ba09cc55fb622b3f41ea5307cd15d2" exitCode=60
Oct 29 22:45:04.135887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.135843    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" event={"ID":"b6184068-b2bf-487b-9b92-200dd0b4fd10","Type":"ContainerDied","Data":"3a7eff867e1393646252b0de103fa14953ba09cc55fb622b3f41ea5307cd15d2"}
Oct 29 22:45:04.136041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.136023    4860 scope.go:122] "RemoveContainer" containerID="3a7eff867e1393646252b0de103fa14953ba09cc55fb622b3f41ea5307cd15d2"
Oct 29 22:45:04.137716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.137694    4860 generic.go:356] "Generic (PLEG): container finished" podID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" containerID="28434ff056e4e2bd86b4053cfabbbd783175e4189c5f509b29d0273143c23432" exitCode=60
Oct 29 22:45:04.137824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.137776    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw" event={"ID":"455b3b28-47ae-40e5-a94d-c9ca25dc15c0","Type":"ContainerDied","Data":"28434ff056e4e2bd86b4053cfabbbd783175e4189c5f509b29d0273143c23432"}
Oct 29 22:45:04.137949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.137933    4860 scope.go:122] "RemoveContainer" containerID="28434ff056e4e2bd86b4053cfabbbd783175e4189c5f509b29d0273143c23432"
Oct 29 22:45:04.144843 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.144819    4860 generic.go:356] "Generic (PLEG): container finished" podID="c2580b08-b5bb-4306-a829-db8b7d832781" containerID="71572e5d961fea33a89fd8a9f1fd0b734b466e31870007e04e94a0157d60c6b3" exitCode=60
Oct 29 22:45:04.145233 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.145218    4860 scope.go:122] "RemoveContainer" containerID="71572e5d961fea33a89fd8a9f1fd0b734b466e31870007e04e94a0157d60c6b3"
Oct 29 22:45:04.145306 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.145253    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4" event={"ID":"c2580b08-b5bb-4306-a829-db8b7d832781","Type":"ContainerDied","Data":"71572e5d961fea33a89fd8a9f1fd0b734b466e31870007e04e94a0157d60c6b3"}
Oct 29 22:45:04.157154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.157133    4860 generic.go:356] "Generic (PLEG): container finished" podID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" containerID="7e967827e10b9479f29579a8e936ab0b153b71adaa397ea1243caa9a08bc129a" exitCode=60
Oct 29 22:45:04.157319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.157302    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj" event={"ID":"1276947c-a4b8-4eb3-8aaf-2d589e81b84c","Type":"ContainerDied","Data":"7e967827e10b9479f29579a8e936ab0b153b71adaa397ea1243caa9a08bc129a"}
Oct 29 22:45:04.157720 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.157704    4860 scope.go:122] "RemoveContainer" containerID="7e967827e10b9479f29579a8e936ab0b153b71adaa397ea1243caa9a08bc129a"
Oct 29 22:45:04.160227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.160203    4860 generic.go:356] "Generic (PLEG): container finished" podID="b032a2d6-c0e0-46cf-9265-e214157142f2" containerID="d54d9935ff0db8e254472c894090d6df9fd898b8098cc8a6b364107f91026f0d" exitCode=60
Oct 29 22:45:04.160368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.160289    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds" event={"ID":"b032a2d6-c0e0-46cf-9265-e214157142f2","Type":"ContainerDied","Data":"d54d9935ff0db8e254472c894090d6df9fd898b8098cc8a6b364107f91026f0d"}
Oct 29 22:45:04.160543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.160528    4860 scope.go:122] "RemoveContainer" containerID="d54d9935ff0db8e254472c894090d6df9fd898b8098cc8a6b364107f91026f0d"
Oct 29 22:45:04.163069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.163046    4860 generic.go:356] "Generic (PLEG): container finished" podID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" containerID="15f3a6a88f8571bbca21646e9422009efe8ffb68d857f07e91fc4ff4121f3acd" exitCode=60
Oct 29 22:45:04.163149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.163082    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc" event={"ID":"f00c15ba-e735-40a2-a1be-f32f20d8d7be","Type":"ContainerDied","Data":"15f3a6a88f8571bbca21646e9422009efe8ffb68d857f07e91fc4ff4121f3acd"}
Oct 29 22:45:04.163386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.163365    4860 scope.go:122] "RemoveContainer" containerID="15f3a6a88f8571bbca21646e9422009efe8ffb68d857f07e91fc4ff4121f3acd"
Oct 29 22:45:04.296842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:04.296809    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:05.181079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.181051    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594" event={"ID":"696b181a-c751-4741-8f95-86a613ef965c","Type":"ContainerStarted","Data":"c62721be183fb3c60a675f7a76a1fac3ce3401fb642ccd1a5df64c823f61cf47"}
Oct 29 22:45:05.185058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.183985    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" event={"ID":"b6184068-b2bf-487b-9b92-200dd0b4fd10","Type":"ContainerStarted","Data":"c2788439c0acf87217aae0960227913dc5ddea49bf07dd5c443944f083dbc0fe"}
Oct 29 22:45:05.188516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.187133    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj" event={"ID":"1276947c-a4b8-4eb3-8aaf-2d589e81b84c","Type":"ContainerStarted","Data":"5dcab0728b537a7159b790c0a1867b22853491b1566024aa7ed4f70cc61bfdc4"}
Oct 29 22:45:05.190992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.189977    4860 generic.go:356] "Generic (PLEG): container finished" podID="2f97838e-68b7-4252-93dd-f535f4cca7f1" containerID="43406f9b774268782c41aeda31f4e7d2544ba63d1ba7cde8fd655c22a84f7b86" exitCode=60
Oct 29 22:45:05.190992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.190087    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w" event={"ID":"2f97838e-68b7-4252-93dd-f535f4cca7f1","Type":"ContainerDied","Data":"43406f9b774268782c41aeda31f4e7d2544ba63d1ba7cde8fd655c22a84f7b86"}
Oct 29 22:45:05.190992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.190134    4860 scope.go:122] "RemoveContainer" containerID="3a9ff3348e45c46d682585ca988d0b027c39d07d500a00e9f9a9460c277e055a"
Oct 29 22:45:05.190992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.190589    4860 scope.go:122] "RemoveContainer" containerID="43406f9b774268782c41aeda31f4e7d2544ba63d1ba7cde8fd655c22a84f7b86"
Oct 29 22:45:05.190992 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:05.190924    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w_odh-ai-gateway-infra(2f97838e-68b7-4252-93dd-f535f4cca7f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1"
Oct 29 22:45:05.194877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.194857    4860 generic.go:356] "Generic (PLEG): container finished" podID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" containerID="2a894177ef66108bb48c355543d6eb51444cee713d3d0ea8cadcef6150ee2d28" exitCode=60
Oct 29 22:45:05.194991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.194932    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767" event={"ID":"6d1c0aa4-106c-44cf-8c66-c94717ba5c48","Type":"ContainerDied","Data":"2a894177ef66108bb48c355543d6eb51444cee713d3d0ea8cadcef6150ee2d28"}
Oct 29 22:45:05.195163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.195147    4860 scope.go:122] "RemoveContainer" containerID="2a894177ef66108bb48c355543d6eb51444cee713d3d0ea8cadcef6150ee2d28"
Oct 29 22:45:05.195481 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:05.195447    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767_odh-ai-gateway-infra(6d1c0aa4-106c-44cf-8c66-c94717ba5c48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48"
Oct 29 22:45:05.196982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.196959    4860 generic.go:356] "Generic (PLEG): container finished" podID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" containerID="e03cfbb3ca7b333de7252ac7a680cb60bf70636838d55eb907986f3be145d93b" exitCode=60
Oct 29 22:45:05.197070 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.197039    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng" event={"ID":"793ee6ac-d14d-480f-93d2-32bba5a4aed5","Type":"ContainerDied","Data":"e03cfbb3ca7b333de7252ac7a680cb60bf70636838d55eb907986f3be145d93b"}
Oct 29 22:45:05.197299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.197281    4860 scope.go:122] "RemoveContainer" containerID="e03cfbb3ca7b333de7252ac7a680cb60bf70636838d55eb907986f3be145d93b"
Oct 29 22:45:05.197619 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:05.197593    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng_odh-ai-gateway-infra(793ee6ac-d14d-480f-93d2-32bba5a4aed5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5"
Oct 29 22:45:05.199418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.199395    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds" event={"ID":"b032a2d6-c0e0-46cf-9265-e214157142f2","Type":"ContainerStarted","Data":"31ee2f10b5234f1e0adab086d831324a113682dc3f0de8eb966b6c22143ca707"}
Oct 29 22:45:05.201792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.201767    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc" event={"ID":"f00c15ba-e735-40a2-a1be-f32f20d8d7be","Type":"ContainerStarted","Data":"d90b5955ea1d29ca48e385fb047b2ae81ec30c3ce9536d5d86719c87fab07979"}
Oct 29 22:45:05.204046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.204022    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk" event={"ID":"8632936a-6257-4b14-9607-d7b18dfb85f7","Type":"ContainerStarted","Data":"0157b190ec11bcca313fcae71de4c1a068dd9f21280e3be64d110be794263ed2"}
Oct 29 22:45:05.206077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.206050    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs" event={"ID":"46509b71-e375-40ca-9a04-d35ddaa6e8bc","Type":"ContainerStarted","Data":"cedd6aa0b6951f0d51ab12e43390fa30cd3975b796679eeb832d29ea88db412f"}
Oct 29 22:45:05.208159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.208128    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2" event={"ID":"3b8a9038-c7c7-497b-b177-8c8342af29a4","Type":"ContainerStarted","Data":"d8bc58ff8db7cb0e98df1537328f8c9d620ccc90e345dc2648b92638a3f5bf9f"}
Oct 29 22:45:05.208787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.208765    4860 scope.go:122] "RemoveContainer" containerID="2735260ee7131f54d786ecfbaa9b4b9e782287847fc10973e21cc881b62820aa"
Oct 29 22:45:05.223805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:05.223781    4860 scope.go:122] "RemoveContainer" containerID="8ac038aba14417f4b7ad721d15f5b1d4271862ace9e38f62d82e571ca79e776b"
Oct 29 22:45:06.205230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.205178    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:06.227102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.227072    4860 generic.go:356] "Generic (PLEG): container finished" podID="41a54691-ca33-41d5-bdf2-f3d603727ee3" containerID="d7a5884366c9cb4517b759fe6bd358ea13ccfcd8783de119839e1dfb6115b48f" exitCode=60
Oct 29 22:45:06.227272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.227157    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns" event={"ID":"41a54691-ca33-41d5-bdf2-f3d603727ee3","Type":"ContainerDied","Data":"d7a5884366c9cb4517b759fe6bd358ea13ccfcd8783de119839e1dfb6115b48f"}
Oct 29 22:45:06.227272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.227221    4860 scope.go:122] "RemoveContainer" containerID="fed0e151ebd088a92c5f686e467d7a31a01532fa627616180a6d756661e7fe44"
Oct 29 22:45:06.227570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.227549    4860 scope.go:122] "RemoveContainer" containerID="d7a5884366c9cb4517b759fe6bd358ea13ccfcd8783de119839e1dfb6115b48f"
Oct 29 22:45:06.228032 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.228003    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns_odh-ai-gateway-infra(41a54691-ca33-41d5-bdf2-f3d603727ee3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3"
Oct 29 22:45:06.229870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.229849    4860 generic.go:356] "Generic (PLEG): container finished" podID="8632936a-6257-4b14-9607-d7b18dfb85f7" containerID="0157b190ec11bcca313fcae71de4c1a068dd9f21280e3be64d110be794263ed2" exitCode=60
Oct 29 22:45:06.229984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.229946    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk" event={"ID":"8632936a-6257-4b14-9607-d7b18dfb85f7","Type":"ContainerDied","Data":"0157b190ec11bcca313fcae71de4c1a068dd9f21280e3be64d110be794263ed2"}
Oct 29 22:45:06.230306 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.230288    4860 scope.go:122] "RemoveContainer" containerID="0157b190ec11bcca313fcae71de4c1a068dd9f21280e3be64d110be794263ed2"
Oct 29 22:45:06.230793 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.230772    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk_odh-ai-gateway-infra(8632936a-6257-4b14-9607-d7b18dfb85f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk" podUID="8632936a-6257-4b14-9607-d7b18dfb85f7"
Oct 29 22:45:06.235545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.235522    4860 generic.go:356] "Generic (PLEG): container finished" podID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" containerID="cedd6aa0b6951f0d51ab12e43390fa30cd3975b796679eeb832d29ea88db412f" exitCode=60
Oct 29 22:45:06.235673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.235615    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs" event={"ID":"46509b71-e375-40ca-9a04-d35ddaa6e8bc","Type":"ContainerDied","Data":"cedd6aa0b6951f0d51ab12e43390fa30cd3975b796679eeb832d29ea88db412f"}
Oct 29 22:45:06.235923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.235901    4860 scope.go:122] "RemoveContainer" containerID="cedd6aa0b6951f0d51ab12e43390fa30cd3975b796679eeb832d29ea88db412f"
Oct 29 22:45:06.236310 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.236286    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs_odh-ai-gateway-infra(46509b71-e375-40ca-9a04-d35ddaa6e8bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc"
Oct 29 22:45:06.238280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.238257    4860 generic.go:356] "Generic (PLEG): container finished" podID="3b8a9038-c7c7-497b-b177-8c8342af29a4" containerID="d8bc58ff8db7cb0e98df1537328f8c9d620ccc90e345dc2648b92638a3f5bf9f" exitCode=60
Oct 29 22:45:06.238428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.238408    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2" event={"ID":"3b8a9038-c7c7-497b-b177-8c8342af29a4","Type":"ContainerDied","Data":"d8bc58ff8db7cb0e98df1537328f8c9d620ccc90e345dc2648b92638a3f5bf9f"}
Oct 29 22:45:06.238853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.238831    4860 scope.go:122] "RemoveContainer" containerID="d8bc58ff8db7cb0e98df1537328f8c9d620ccc90e345dc2648b92638a3f5bf9f"
Oct 29 22:45:06.243449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.243412    4860 generic.go:356] "Generic (PLEG): container finished" podID="696b181a-c751-4741-8f95-86a613ef965c" containerID="c62721be183fb3c60a675f7a76a1fac3ce3401fb642ccd1a5df64c823f61cf47" exitCode=60
Oct 29 22:45:06.243572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.243496    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594" event={"ID":"696b181a-c751-4741-8f95-86a613ef965c","Type":"ContainerDied","Data":"c62721be183fb3c60a675f7a76a1fac3ce3401fb642ccd1a5df64c823f61cf47"}
Oct 29 22:45:06.243765 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.243745    4860 scope.go:122] "RemoveContainer" containerID="c62721be183fb3c60a675f7a76a1fac3ce3401fb642ccd1a5df64c823f61cf47"
Oct 29 22:45:06.244184 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.244148    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594_odh-ai-gateway-infra(696b181a-c751-4741-8f95-86a613ef965c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594" podUID="696b181a-c751-4741-8f95-86a613ef965c"
Oct 29 22:45:06.245949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.245928    4860 generic.go:356] "Generic (PLEG): container finished" podID="b6184068-b2bf-487b-9b92-200dd0b4fd10" containerID="c2788439c0acf87217aae0960227913dc5ddea49bf07dd5c443944f083dbc0fe" exitCode=60
Oct 29 22:45:06.246034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.246008    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" event={"ID":"b6184068-b2bf-487b-9b92-200dd0b4fd10","Type":"ContainerDied","Data":"c2788439c0acf87217aae0960227913dc5ddea49bf07dd5c443944f083dbc0fe"}
Oct 29 22:45:06.246305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.246286    4860 scope.go:122] "RemoveContainer" containerID="c2788439c0acf87217aae0960227913dc5ddea49bf07dd5c443944f083dbc0fe"
Oct 29 22:45:06.246703 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.246677    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78_odh-ai-gateway-infra(b6184068-b2bf-487b-9b92-200dd0b4fd10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10"
Oct 29 22:45:06.246968 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.246930    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2_odh-ai-gateway-infra(3b8a9038-c7c7-497b-b177-8c8342af29a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4"
Oct 29 22:45:06.248411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.248390    4860 generic.go:356] "Generic (PLEG): container finished" podID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" containerID="09f31566232b809306f4cdd8cf6b660b816c79a97de02546683838e9f0b377a2" exitCode=60
Oct 29 22:45:06.248509 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.248479    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw" event={"ID":"455b3b28-47ae-40e5-a94d-c9ca25dc15c0","Type":"ContainerDied","Data":"09f31566232b809306f4cdd8cf6b660b816c79a97de02546683838e9f0b377a2"}
Oct 29 22:45:06.248751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.248730    4860 scope.go:122] "RemoveContainer" containerID="09f31566232b809306f4cdd8cf6b660b816c79a97de02546683838e9f0b377a2"
Oct 29 22:45:06.249089 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.249065    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw_odh-ai-gateway-infra(455b3b28-47ae-40e5-a94d-c9ca25dc15c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0"
Oct 29 22:45:06.250665 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.250641    4860 generic.go:356] "Generic (PLEG): container finished" podID="c2580b08-b5bb-4306-a829-db8b7d832781" containerID="d1c465560eba00d57ecdb973763a81e2bafc31748cbf1076bbec6a14aa1c3dc8" exitCode=60
Oct 29 22:45:06.250784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.250762    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4" event={"ID":"c2580b08-b5bb-4306-a829-db8b7d832781","Type":"ContainerDied","Data":"d1c465560eba00d57ecdb973763a81e2bafc31748cbf1076bbec6a14aa1c3dc8"}
Oct 29 22:45:06.250960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.250945    4860 scope.go:122] "RemoveContainer" containerID="d1c465560eba00d57ecdb973763a81e2bafc31748cbf1076bbec6a14aa1c3dc8"
Oct 29 22:45:06.251230 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.251212    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4_odh-ai-gateway-infra(c2580b08-b5bb-4306-a829-db8b7d832781)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4" podUID="c2580b08-b5bb-4306-a829-db8b7d832781"
Oct 29 22:45:06.253179 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.253157    4860 generic.go:356] "Generic (PLEG): container finished" podID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" containerID="5dcab0728b537a7159b790c0a1867b22853491b1566024aa7ed4f70cc61bfdc4" exitCode=60
Oct 29 22:45:06.253273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.253235    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj" event={"ID":"1276947c-a4b8-4eb3-8aaf-2d589e81b84c","Type":"ContainerDied","Data":"5dcab0728b537a7159b790c0a1867b22853491b1566024aa7ed4f70cc61bfdc4"}
Oct 29 22:45:06.253500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.253484    4860 scope.go:122] "RemoveContainer" containerID="5dcab0728b537a7159b790c0a1867b22853491b1566024aa7ed4f70cc61bfdc4"
Oct 29 22:45:06.253844 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.253826    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj_odh-ai-gateway-infra(1276947c-a4b8-4eb3-8aaf-2d589e81b84c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c"
Oct 29 22:45:06.256965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.256943    4860 scope.go:122] "RemoveContainer" containerID="43406f9b774268782c41aeda31f4e7d2544ba63d1ba7cde8fd655c22a84f7b86"
Oct 29 22:45:06.257246 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.257221    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w_odh-ai-gateway-infra(2f97838e-68b7-4252-93dd-f535f4cca7f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1"
Oct 29 22:45:06.259069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.259052    4860 scope.go:122] "RemoveContainer" containerID="2a894177ef66108bb48c355543d6eb51444cee713d3d0ea8cadcef6150ee2d28"
Oct 29 22:45:06.259416 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.259390    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767_odh-ai-gateway-infra(6d1c0aa4-106c-44cf-8c66-c94717ba5c48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48"
Oct 29 22:45:06.261337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.261305    4860 scope.go:122] "RemoveContainer" containerID="e03cfbb3ca7b333de7252ac7a680cb60bf70636838d55eb907986f3be145d93b"
Oct 29 22:45:06.261630 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.261608    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng_odh-ai-gateway-infra(793ee6ac-d14d-480f-93d2-32bba5a4aed5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5"
Oct 29 22:45:06.263512 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.263488    4860 generic.go:356] "Generic (PLEG): container finished" podID="b032a2d6-c0e0-46cf-9265-e214157142f2" containerID="31ee2f10b5234f1e0adab086d831324a113682dc3f0de8eb966b6c22143ca707" exitCode=60
Oct 29 22:45:06.263618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.263570    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds" event={"ID":"b032a2d6-c0e0-46cf-9265-e214157142f2","Type":"ContainerDied","Data":"31ee2f10b5234f1e0adab086d831324a113682dc3f0de8eb966b6c22143ca707"}
Oct 29 22:45:06.263956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.263939    4860 scope.go:122] "RemoveContainer" containerID="31ee2f10b5234f1e0adab086d831324a113682dc3f0de8eb966b6c22143ca707"
Oct 29 22:45:06.264217 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.264193    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds_odh-ai-gateway-infra(b032a2d6-c0e0-46cf-9265-e214157142f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds" podUID="b032a2d6-c0e0-46cf-9265-e214157142f2"
Oct 29 22:45:06.265934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.265911    4860 generic.go:356] "Generic (PLEG): container finished" podID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" containerID="d90b5955ea1d29ca48e385fb047b2ae81ec30c3ce9536d5d86719c87fab07979" exitCode=60
Oct 29 22:45:06.266038 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.265930    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc" event={"ID":"f00c15ba-e735-40a2-a1be-f32f20d8d7be","Type":"ContainerDied","Data":"d90b5955ea1d29ca48e385fb047b2ae81ec30c3ce9536d5d86719c87fab07979"}
Oct 29 22:45:06.266282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.266262    4860 scope.go:122] "RemoveContainer" containerID="d90b5955ea1d29ca48e385fb047b2ae81ec30c3ce9536d5d86719c87fab07979"
Oct 29 22:45:06.266567 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.266542    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc_odh-ai-gateway-infra(f00c15ba-e735-40a2-a1be-f32f20d8d7be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be"
Oct 29 22:45:06.271355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.271318    4860 generic.go:356] "Generic (PLEG): container finished" podID="80f7cc19-3ccf-4e4a-8d30-8f76251885b0" containerID="2a2c8d15316f29e1f163fbc4cf49f19de17d4febd1c5ed76a4d7c9e5b3c44849" exitCode=60
Oct 29 22:45:06.271471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.271369    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5" event={"ID":"80f7cc19-3ccf-4e4a-8d30-8f76251885b0","Type":"ContainerDied","Data":"2a2c8d15316f29e1f163fbc4cf49f19de17d4febd1c5ed76a4d7c9e5b3c44849"}
Oct 29 22:45:06.271745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.271727    4860 scope.go:122] "RemoveContainer" containerID="2a2c8d15316f29e1f163fbc4cf49f19de17d4febd1c5ed76a4d7c9e5b3c44849"
Oct 29 22:45:06.272031 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:06.272007    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5_odh-ai-gateway-infra(80f7cc19-3ccf-4e4a-8d30-8f76251885b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5" podUID="80f7cc19-3ccf-4e4a-8d30-8f76251885b0"
Oct 29 22:45:06.305233 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.305208    4860 scope.go:122] "RemoveContainer" containerID="3f2d5ddd05d98b2e7007d1aac07d20432bf9231798fa84556f6d28d183b15ce8"
Oct 29 22:45:06.322226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.322204    4860 scope.go:122] "RemoveContainer" containerID="91eb9193cd7d73033533db96362ab02079ae9ebab54b82f8c504655626c0f6ae"
Oct 29 22:45:06.337365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.337343    4860 scope.go:122] "RemoveContainer" containerID="c43dcf1a42c87ea62ee46d9aa634df84bc234994fcf4579ffc8c96e3c14c4fda"
Oct 29 22:45:06.351970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.351949    4860 scope.go:122] "RemoveContainer" containerID="240fa60e0b0dd8a12e4da16b1be47f0a918d73d082aa06196ddcaae422be67fb"
Oct 29 22:45:06.368521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.368503    4860 scope.go:122] "RemoveContainer" containerID="3a7eff867e1393646252b0de103fa14953ba09cc55fb622b3f41ea5307cd15d2"
Oct 29 22:45:06.383134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.383116    4860 scope.go:122] "RemoveContainer" containerID="28434ff056e4e2bd86b4053cfabbbd783175e4189c5f509b29d0273143c23432"
Oct 29 22:45:06.397310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.397291    4860 scope.go:122] "RemoveContainer" containerID="71572e5d961fea33a89fd8a9f1fd0b734b466e31870007e04e94a0157d60c6b3"
Oct 29 22:45:06.412405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.412383    4860 scope.go:122] "RemoveContainer" containerID="7e967827e10b9479f29579a8e936ab0b153b71adaa397ea1243caa9a08bc129a"
Oct 29 22:45:06.426785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.426768    4860 scope.go:122] "RemoveContainer" containerID="d54d9935ff0db8e254472c894090d6df9fd898b8098cc8a6b364107f91026f0d"
Oct 29 22:45:06.441255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.441235    4860 scope.go:122] "RemoveContainer" containerID="15f3a6a88f8571bbca21646e9422009efe8ffb68d857f07e91fc4ff4121f3acd"
Oct 29 22:45:06.456551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:06.456534    4860 scope.go:122] "RemoveContainer" containerID="b06b10da5de40b33bc3bd7d3b1c60e04a2438a7603656cdc2369b2d865f3b54b"
Oct 29 22:45:07.283812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.283788    4860 scope.go:122] "RemoveContainer" containerID="0157b190ec11bcca313fcae71de4c1a068dd9f21280e3be64d110be794263ed2"
Oct 29 22:45:07.284260 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:07.284108    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk_odh-ai-gateway-infra(8632936a-6257-4b14-9607-d7b18dfb85f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk" podUID="8632936a-6257-4b14-9607-d7b18dfb85f7"
Oct 29 22:45:07.285589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.285572    4860 scope.go:122] "RemoveContainer" containerID="cedd6aa0b6951f0d51ab12e43390fa30cd3975b796679eeb832d29ea88db412f"
Oct 29 22:45:07.285869 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:07.285845    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs_odh-ai-gateway-infra(46509b71-e375-40ca-9a04-d35ddaa6e8bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc"
Oct 29 22:45:07.287247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.287233    4860 scope.go:122] "RemoveContainer" containerID="d8bc58ff8db7cb0e98df1537328f8c9d620ccc90e345dc2648b92638a3f5bf9f"
Oct 29 22:45:07.287492 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:07.287469    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2_odh-ai-gateway-infra(3b8a9038-c7c7-497b-b177-8c8342af29a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4"
Oct 29 22:45:07.288881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.288862    4860 scope.go:122] "RemoveContainer" containerID="c62721be183fb3c60a675f7a76a1fac3ce3401fb642ccd1a5df64c823f61cf47"
Oct 29 22:45:07.289125 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:07.289105    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594_odh-ai-gateway-infra(696b181a-c751-4741-8f95-86a613ef965c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594" podUID="696b181a-c751-4741-8f95-86a613ef965c"
Oct 29 22:45:07.290741 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.290718    4860 scope.go:122] "RemoveContainer" containerID="c2788439c0acf87217aae0960227913dc5ddea49bf07dd5c443944f083dbc0fe"
Oct 29 22:45:07.290971 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:07.290947    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78_odh-ai-gateway-infra(b6184068-b2bf-487b-9b92-200dd0b4fd10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10"
Oct 29 22:45:07.292457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.292439    4860 scope.go:122] "RemoveContainer" containerID="09f31566232b809306f4cdd8cf6b660b816c79a97de02546683838e9f0b377a2"
Oct 29 22:45:07.292717 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:07.292694    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw_odh-ai-gateway-infra(455b3b28-47ae-40e5-a94d-c9ca25dc15c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0"
Oct 29 22:45:07.294005 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.293987    4860 scope.go:122] "RemoveContainer" containerID="d1c465560eba00d57ecdb973763a81e2bafc31748cbf1076bbec6a14aa1c3dc8"
Oct 29 22:45:07.294227 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:07.294205    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4_odh-ai-gateway-infra(c2580b08-b5bb-4306-a829-db8b7d832781)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4" podUID="c2580b08-b5bb-4306-a829-db8b7d832781"
Oct 29 22:45:07.295608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.295590    4860 scope.go:122] "RemoveContainer" containerID="5dcab0728b537a7159b790c0a1867b22853491b1566024aa7ed4f70cc61bfdc4"
Oct 29 22:45:07.295822 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:07.295799    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj_odh-ai-gateway-infra(1276947c-a4b8-4eb3-8aaf-2d589e81b84c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c"
Oct 29 22:45:07.297372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.297352    4860 scope.go:122] "RemoveContainer" containerID="31ee2f10b5234f1e0adab086d831324a113682dc3f0de8eb966b6c22143ca707"
Oct 29 22:45:07.297617 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:07.297597    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds_odh-ai-gateway-infra(b032a2d6-c0e0-46cf-9265-e214157142f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds" podUID="b032a2d6-c0e0-46cf-9265-e214157142f2"
Oct 29 22:45:07.299054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.299035    4860 scope.go:122] "RemoveContainer" containerID="d90b5955ea1d29ca48e385fb047b2ae81ec30c3ce9536d5d86719c87fab07979"
Oct 29 22:45:07.299274 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:07.299251    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc_odh-ai-gateway-infra(f00c15ba-e735-40a2-a1be-f32f20d8d7be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be"
Oct 29 22:45:07.300981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.300962    4860 scope.go:122] "RemoveContainer" containerID="2a2c8d15316f29e1f163fbc4cf49f19de17d4febd1c5ed76a4d7c9e5b3c44849"
Oct 29 22:45:07.301214 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:07.301190    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5_odh-ai-gateway-infra(80f7cc19-3ccf-4e4a-8d30-8f76251885b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5" podUID="80f7cc19-3ccf-4e4a-8d30-8f76251885b0"
Oct 29 22:45:07.302699 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.302683    4860 scope.go:122] "RemoveContainer" containerID="d7a5884366c9cb4517b759fe6bd358ea13ccfcd8783de119839e1dfb6115b48f"
Oct 29 22:45:07.302889 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:07.302873    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns_odh-ai-gateway-infra(41a54691-ca33-41d5-bdf2-f3d603727ee3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3"
Oct 29 22:45:07.681027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.680990    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:07.893085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:07.893054    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:09.270616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.270567    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"]
Oct 29 22:45:09.271438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.271401    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm" podUID="c5f2142b-4344-47d4-bdf0-40a45e61bb3e" containerName="maas-api" containerID="cri-o://7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395" gracePeriod=30
Oct 29 22:45:09.348393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.348360    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:09.354838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.354796    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"]
Oct 29 22:45:09.647177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.646774    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"
Oct 29 22:45:09.769431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.761293    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-kube-api-access-gxbzr\" (UniqueName: \"kubernetes.io/projected/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-kube-api-access-gxbzr\") pod \"80f7cc19-3ccf-4e4a-8d30-8f76251885b0\" (UID: \"80f7cc19-3ccf-4e4a-8d30-8f76251885b0\") "
Oct 29 22:45:09.769431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.761421    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-maas-api-service-ca\") pod \"80f7cc19-3ccf-4e4a-8d30-8f76251885b0\" (UID: \"80f7cc19-3ccf-4e4a-8d30-8f76251885b0\") "
Oct 29 22:45:09.789236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.772818    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-maas-api-service-ca" pod "80f7cc19-3ccf-4e4a-8d30-8f76251885b0" (UID: "80f7cc19-3ccf-4e4a-8d30-8f76251885b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:09.864374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.864256    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:09.950375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.939381    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-kube-api-access-gxbzr" pod "80f7cc19-3ccf-4e4a-8d30-8f76251885b0" (UID: "80f7cc19-3ccf-4e4a-8d30-8f76251885b0"). InnerVolumeSpecName "kube-api-access-gxbzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:09.950375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.942263    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:09.966269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.966227    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxbzr\" (UniqueName: \"kubernetes.io/projected/80f7cc19-3ccf-4e4a-8d30-8f76251885b0-kube-api-access-gxbzr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:09.972105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:09.972077    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:45:10.067411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.067379    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-maas-api-tls\") pod \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") "
Oct 29 22:45:10.067613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.067512    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-metrics-certs\") pod \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") "
Oct 29 22:45:10.067682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.067621    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-openshift-service-ca\") pod \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") "
Oct 29 22:45:10.067880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.067861    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-kube-api-access-c2jrm\" (UniqueName: \"kubernetes.io/projected/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-kube-api-access-c2jrm\") pod \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\" (UID: \"c5f2142b-4344-47d4-bdf0-40a45e61bb3e\") "
Oct 29 22:45:10.068170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.068144    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-openshift-service-ca" pod "c5f2142b-4344-47d4-bdf0-40a45e61bb3e" (UID: "c5f2142b-4344-47d4-bdf0-40a45e61bb3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:10.068927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.068893    4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-openshift-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:10.069864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.069840    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-maas-api-tls" pod "c5f2142b-4344-47d4-bdf0-40a45e61bb3e" (UID: "c5f2142b-4344-47d4-bdf0-40a45e61bb3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:45:10.070156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.070135    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-metrics-certs" pod "c5f2142b-4344-47d4-bdf0-40a45e61bb3e" (UID: "c5f2142b-4344-47d4-bdf0-40a45e61bb3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:45:10.070858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.070839    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-kube-api-access-c2jrm" pod "c5f2142b-4344-47d4-bdf0-40a45e61bb3e" (UID: "c5f2142b-4344-47d4-bdf0-40a45e61bb3e"). InnerVolumeSpecName "kube-api-access-c2jrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:10.178775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.170576    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"]
Oct 29 22:45:10.178775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.171597    4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-metrics-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:10.178775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.171621    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2jrm\" (UniqueName: \"kubernetes.io/projected/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-kube-api-access-c2jrm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:10.178775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.171635    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f2142b-4344-47d4-bdf0-40a45e61bb3e-maas-api-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:10.186411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.180846    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4d3168-qqz7p"]
Oct 29 22:45:10.224414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.224314    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:10.358010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.357972    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5" event={"ID":"80f7cc19-3ccf-4e4a-8d30-8f76251885b0","Type":"ContainerDied","Data":"50c2b140247fc99da067615a10cd296a63ea5a13a135e584e3ddf4208a3620dc"}
Oct 29 22:45:10.358010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.357994    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"
Oct 29 22:45:10.358552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.358044    4860 scope.go:122] "RemoveContainer" containerID="2a2c8d15316f29e1f163fbc4cf49f19de17d4febd1c5ed76a4d7c9e5b3c44849"
Oct 29 22:45:10.360420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.360393    4860 generic.go:356] "Generic (PLEG): container finished" podID="c5f2142b-4344-47d4-bdf0-40a45e61bb3e" containerID="7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395" exitCode=0
Oct 29 22:45:10.360550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.360442    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm" event={"ID":"c5f2142b-4344-47d4-bdf0-40a45e61bb3e","Type":"ContainerDied","Data":"7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395"}
Oct 29 22:45:10.360550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.360463    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"
Oct 29 22:45:10.360550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.360483    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm" event={"ID":"c5f2142b-4344-47d4-bdf0-40a45e61bb3e","Type":"ContainerDied","Data":"dd780bab27632d5ed82cf66fc9a35f0820c84588dfdfdea198fdb083d96494b4"}
Oct 29 22:45:10.373524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.373498    4860 scope.go:122] "RemoveContainer" containerID="7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395"
Oct 29 22:45:10.387647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.387628    4860 scope.go:122] "RemoveContainer" containerID="7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395"
Oct 29 22:45:10.387958 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:10.387939    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395\": container with ID starting with 7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395 not found: ID does not exist" containerID="7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395"
Oct 29 22:45:10.388040 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.387980    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395"} err="failed to get container status \"7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395\": rpc error: code = NotFound desc = could not find container \"7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395\": container with ID starting with 7bd35463f053037174acaa6eedeb80d69d87652d75e75325da4210eccdd62395 not found: ID does not exist"
Oct 29 22:45:10.418605 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.418584    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"]
Oct 29 22:45:10.424197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.424144    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d3168-5785964b99-wntgm"]
Oct 29 22:45:10.470929 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.470894    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"]
Oct 29 22:45:10.476562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.476534    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4d3168-29888565-4mzc5"]
Oct 29 22:45:10.670335 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:10.670305    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:11.828528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:11.828494    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:12.115712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:12.115634    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f835847-daae-471e-a710-84ee468eff27" path="/var/lib/kubelet/pods/5f835847-daae-471e-a710-84ee468eff27/volumes"
Oct 29 22:45:12.116124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:12.116110    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f7cc19-3ccf-4e4a-8d30-8f76251885b0" path="/var/lib/kubelet/pods/80f7cc19-3ccf-4e4a-8d30-8f76251885b0/volumes"
Oct 29 22:45:12.116609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:12.116595    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f2142b-4344-47d4-bdf0-40a45e61bb3e" path="/var/lib/kubelet/pods/c5f2142b-4344-47d4-bdf0-40a45e61bb3e/volumes"
Oct 29 22:45:12.769785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:12.769750    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:13.643860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:13.643825    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:15.235686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:15.235639    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:16.105136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:16.105107    4860 scope.go:122] "RemoveContainer" containerID="5dcab0728b537a7159b790c0a1867b22853491b1566024aa7ed4f70cc61bfdc4"
Oct 29 22:45:16.105332 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:16.105188    4860 scope.go:122] "RemoveContainer" containerID="e03cfbb3ca7b333de7252ac7a680cb60bf70636838d55eb907986f3be145d93b"
Oct 29 22:45:16.916072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:16.916039    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:17.104993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:17.104958    4860 scope.go:122] "RemoveContainer" containerID="cedd6aa0b6951f0d51ab12e43390fa30cd3975b796679eeb832d29ea88db412f"
Oct 29 22:45:17.105200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:17.105168    4860 scope.go:122] "RemoveContainer" containerID="d1c465560eba00d57ecdb973763a81e2bafc31748cbf1076bbec6a14aa1c3dc8"
Oct 29 22:45:17.105482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:17.105457    4860 scope.go:122] "RemoveContainer" containerID="31ee2f10b5234f1e0adab086d831324a113682dc3f0de8eb966b6c22143ca707"
Oct 29 22:45:17.105649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:17.105624    4860 scope.go:122] "RemoveContainer" containerID="c62721be183fb3c60a675f7a76a1fac3ce3401fb642ccd1a5df64c823f61cf47"
Oct 29 22:45:17.439493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:17.439462    4860 generic.go:356] "Generic (PLEG): container finished" podID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" containerID="8b10bd2f1e8d73fc4b6aa67b792ce5672f8bd307d4b38352ab0cae697da0bbe4" exitCode=60
Oct 29 22:45:17.439666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:17.439536    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj" event={"ID":"1276947c-a4b8-4eb3-8aaf-2d589e81b84c","Type":"ContainerDied","Data":"8b10bd2f1e8d73fc4b6aa67b792ce5672f8bd307d4b38352ab0cae697da0bbe4"}
Oct 29 22:45:17.439666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:17.439593    4860 scope.go:122] "RemoveContainer" containerID="5dcab0728b537a7159b790c0a1867b22853491b1566024aa7ed4f70cc61bfdc4"
Oct 29 22:45:17.440064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:17.440043    4860 scope.go:122] "RemoveContainer" containerID="8b10bd2f1e8d73fc4b6aa67b792ce5672f8bd307d4b38352ab0cae697da0bbe4"
Oct 29 22:45:17.440547 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:17.440522    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj_odh-ai-gateway-infra(1276947c-a4b8-4eb3-8aaf-2d589e81b84c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c"
Oct 29 22:45:17.442146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:17.442125    4860 generic.go:356] "Generic (PLEG): container finished" podID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" containerID="d7b4d5b12f5aef941e65e00a9c098e64427cb32a7ded96b29e4e581532b9424e" exitCode=60
Oct 29 22:45:17.442270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:17.442163    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng" event={"ID":"793ee6ac-d14d-480f-93d2-32bba5a4aed5","Type":"ContainerDied","Data":"d7b4d5b12f5aef941e65e00a9c098e64427cb32a7ded96b29e4e581532b9424e"}
Oct 29 22:45:17.442632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:17.442615    4860 scope.go:122] "RemoveContainer" containerID="d7b4d5b12f5aef941e65e00a9c098e64427cb32a7ded96b29e4e581532b9424e"
Oct 29 22:45:17.442995 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:17.442975    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng_odh-ai-gateway-infra(793ee6ac-d14d-480f-93d2-32bba5a4aed5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5"
Oct 29 22:45:17.455369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:17.455317    4860 scope.go:122] "RemoveContainer" containerID="e03cfbb3ca7b333de7252ac7a680cb60bf70636838d55eb907986f3be145d93b"
Oct 29 22:45:18.063270 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:18.063179    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:45:18.110641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:18.110609    4860 scope.go:122] "RemoveContainer" containerID="c2788439c0acf87217aae0960227913dc5ddea49bf07dd5c443944f083dbc0fe"
Oct 29 22:45:18.110869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:18.110770    4860 scope.go:122] "RemoveContainer" containerID="0157b190ec11bcca313fcae71de4c1a068dd9f21280e3be64d110be794263ed2"
Oct 29 22:45:18.717136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:18.717091    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"]
Oct 29 22:45:18.819553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:18.799001    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:18.882581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:18.882538    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"]
Oct 29 22:45:18.968496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:18.960469    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:19.157179 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:19.157141    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds" event={"ID":"b032a2d6-c0e0-46cf-9265-e214157142f2","Type":"ContainerStarted","Data":"4588af9a72c9a03212fef8405c6d480257c966623caa28e39896987d6097e4f3"}
Oct 29 22:45:19.184233 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:19.182759    4860 scope.go:122] "RemoveContainer" containerID="09f31566232b809306f4cdd8cf6b660b816c79a97de02546683838e9f0b377a2"
Oct 29 22:45:19.251493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:19.251459    4860 scope.go:122] "RemoveContainer" containerID="2a894177ef66108bb48c355543d6eb51444cee713d3d0ea8cadcef6150ee2d28"
Oct 29 22:45:19.259444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:19.252138    4860 scope.go:122] "RemoveContainer" containerID="d7a5884366c9cb4517b759fe6bd358ea13ccfcd8783de119839e1dfb6115b48f"
Oct 29 22:45:19.374353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:19.339917    4860 scope.go:122] "RemoveContainer" containerID="43406f9b774268782c41aeda31f4e7d2544ba63d1ba7cde8fd655c22a84f7b86"
Oct 29 22:45:19.547063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:19.547012    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs" event={"ID":"46509b71-e375-40ca-9a04-d35ddaa6e8bc","Type":"ContainerStarted","Data":"5ebae55fbd731c61488c40d3cf6d415ea0d778d0a8ca9e86dee7e8d0584ad0d9"}
Oct 29 22:45:19.550679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:19.550651    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594" event={"ID":"696b181a-c751-4741-8f95-86a613ef965c","Type":"ContainerStarted","Data":"54998d1e3333404692db7f3339f71955bd8e8723c16f4cb9230ea5f65582b1cd"}
Oct 29 22:45:19.772370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:19.763285    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:19.772370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:19.765097    4860 scope.go:122] "RemoveContainer" containerID="54998d1e3333404692db7f3339f71955bd8e8723c16f4cb9230ea5f65582b1cd"
Oct 29 22:45:19.781681 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:19.766314    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594_odh-ai-gateway-infra(696b181a-c751-4741-8f95-86a613ef965c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594" podUID="696b181a-c751-4741-8f95-86a613ef965c"
Oct 29 22:45:19.804351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:19.782702    4860 scope.go:122] "RemoveContainer" containerID="5ebae55fbd731c61488c40d3cf6d415ea0d778d0a8ca9e86dee7e8d0584ad0d9"
Oct 29 22:45:19.805857 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:19.805817    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs_odh-ai-gateway-infra(46509b71-e375-40ca-9a04-d35ddaa6e8bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc"
Oct 29 22:45:19.823393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:19.790253    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:45:19.975846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:19.973860    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:45:20.080138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.080047    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:20.122510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.090413    4860 generic.go:356] "Generic (PLEG): container finished" podID="c2580b08-b5bb-4306-a829-db8b7d832781" containerID="14f14a79d27a92db301a45afc0aaa21c71b4963e37b96b44b97d0b44c3b6348e" exitCode=60
Oct 29 22:45:20.122510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.090465    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4" event={"ID":"c2580b08-b5bb-4306-a829-db8b7d832781","Type":"ContainerDied","Data":"14f14a79d27a92db301a45afc0aaa21c71b4963e37b96b44b97d0b44c3b6348e"}
Oct 29 22:45:20.122510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.090522    4860 scope.go:122] "RemoveContainer" containerID="d1c465560eba00d57ecdb973763a81e2bafc31748cbf1076bbec6a14aa1c3dc8"
Oct 29 22:45:20.122510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.091315    4860 scope.go:122] "RemoveContainer" containerID="14f14a79d27a92db301a45afc0aaa21c71b4963e37b96b44b97d0b44c3b6348e"
Oct 29 22:45:20.122510 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:20.091763    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4_odh-ai-gateway-infra(c2580b08-b5bb-4306-a829-db8b7d832781)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4" podUID="c2580b08-b5bb-4306-a829-db8b7d832781"
Oct 29 22:45:20.122510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.092433    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:20.122510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.104474    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:20.124315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.124287    4860 scope.go:122] "RemoveContainer" containerID="d8bc58ff8db7cb0e98df1537328f8c9d620ccc90e345dc2648b92638a3f5bf9f"
Oct 29 22:45:20.132528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.132502    4860 scope.go:122] "RemoveContainer" containerID="d90b5955ea1d29ca48e385fb047b2ae81ec30c3ce9536d5d86719c87fab07979"
Oct 29 22:45:20.288347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.267625    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"]
Oct 29 22:45:20.323263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.323216    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" podUID="e292294d-64b9-43b7-a0b9-89f63da16cdb" containerName="istio-proxy" containerID="cri-o://b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64" gracePeriod=30
Oct 29 22:45:20.361397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.359471    4860 scope.go:122] "RemoveContainer" containerID="fa68e5a889fcbe21588b82ecffe14f763fab9ac56d69e0b9bcf1e72d7471fe37"
Oct 29 22:45:20.989652 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:20.965416    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:21.308557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.308516    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"
Oct 29 22:45:21.425597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.417698    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-maas-api-service-ca\") pod \"1276947c-a4b8-4eb3-8aaf-2d589e81b84c\" (UID: \"1276947c-a4b8-4eb3-8aaf-2d589e81b84c\") "
Oct 29 22:45:21.425597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.417954    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-kube-api-access-647wd\" (UniqueName: \"kubernetes.io/projected/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-kube-api-access-647wd\") pod \"1276947c-a4b8-4eb3-8aaf-2d589e81b84c\" (UID: \"1276947c-a4b8-4eb3-8aaf-2d589e81b84c\") "
Oct 29 22:45:21.464796 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.464753    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-kube-api-access-647wd" pod "1276947c-a4b8-4eb3-8aaf-2d589e81b84c" (UID: "1276947c-a4b8-4eb3-8aaf-2d589e81b84c"). InnerVolumeSpecName "kube-api-access-647wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:21.466879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.466711    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-maas-api-service-ca" pod "1276947c-a4b8-4eb3-8aaf-2d589e81b84c" (UID: "1276947c-a4b8-4eb3-8aaf-2d589e81b84c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:21.568691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.544611    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"]
Oct 29 22:45:21.626469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.585214    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-647wd\" (UniqueName: \"kubernetes.io/projected/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-kube-api-access-647wd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:21.626469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.585249    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1276947c-a4b8-4eb3-8aaf-2d589e81b84c-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:21.675356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.664911    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:21.684355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.681882    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:21.727535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.726432    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:45:21.757226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.757181    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"]
Oct 29 22:45:21.833463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.832411    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk" event={"ID":"8632936a-6257-4b14-9607-d7b18dfb85f7","Type":"ContainerStarted","Data":"1c8e42d821f197b0b7d4003f3964f474bc063dbfc43eb10544571475227fb7ee"}
Oct 29 22:45:21.874753 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.872052    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"]
Oct 29 22:45:21.896395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.894913    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"]
Oct 29 22:45:21.908369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.908131    4860 generic.go:356] "Generic (PLEG): container finished" podID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" containerID="5ebae55fbd731c61488c40d3cf6d415ea0d778d0a8ca9e86dee7e8d0584ad0d9" exitCode=60
Oct 29 22:45:21.908529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.908440    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs" event={"ID":"46509b71-e375-40ca-9a04-d35ddaa6e8bc","Type":"ContainerDied","Data":"5ebae55fbd731c61488c40d3cf6d415ea0d778d0a8ca9e86dee7e8d0584ad0d9"}
Oct 29 22:45:21.929751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.927554    4860 scope.go:122] "RemoveContainer" containerID="25a30dd7f9d28af72071c8056f997d4019b4bfdf80909231aff6da05533b7563"
Oct 29 22:45:21.936925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.936492    4860 generic.go:356] "Generic (PLEG): container finished" podID="696b181a-c751-4741-8f95-86a613ef965c" containerID="54998d1e3333404692db7f3339f71955bd8e8723c16f4cb9230ea5f65582b1cd" exitCode=60
Oct 29 22:45:21.936925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.936599    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594" event={"ID":"696b181a-c751-4741-8f95-86a613ef965c","Type":"ContainerDied","Data":"54998d1e3333404692db7f3339f71955bd8e8723c16f4cb9230ea5f65582b1cd"}
Oct 29 22:45:21.937359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.937316    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"
Oct 29 22:45:21.940648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.938501    4860 scope.go:122] "RemoveContainer" containerID="cedd6aa0b6951f0d51ab12e43390fa30cd3975b796679eeb832d29ea88db412f"
Oct 29 22:45:21.940648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.940052    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" event={"ID":"b6184068-b2bf-487b-9b92-200dd0b4fd10","Type":"ContainerStarted","Data":"d4bcbcab3aaf1ef0afb2fb7c09b09c149cf4e4eff05a5de91c898898c15e745a"}
Oct 29 22:45:21.940648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.940577    4860 scope.go:122] "RemoveContainer" containerID="d4bcbcab3aaf1ef0afb2fb7c09b09c149cf4e4eff05a5de91c898898c15e745a"
Oct 29 22:45:21.941047 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:21.941015    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78_odh-ai-gateway-infra(b6184068-b2bf-487b-9b92-200dd0b4fd10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10"
Oct 29 22:45:21.942852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.942822    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj" event={"ID":"1276947c-a4b8-4eb3-8aaf-2d589e81b84c","Type":"ContainerDied","Data":"e7f829a0aa574154595a304d66b1a3324fb25ef7a12c21fb1df8bcc2d282611d"}
Oct 29 22:45:21.942959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:21.942921    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"
Oct 29 22:45:22.050527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.050411    4860 scope.go:122] "RemoveContainer" containerID="c62721be183fb3c60a675f7a76a1fac3ce3401fb642ccd1a5df64c823f61cf47"
Oct 29 22:45:22.052989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.050891    4860 generic.go:356] "Generic (PLEG): container finished" podID="b032a2d6-c0e0-46cf-9265-e214157142f2" containerID="4588af9a72c9a03212fef8405c6d480257c966623caa28e39896987d6097e4f3" exitCode=60
Oct 29 22:45:22.052989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.051444    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds" event={"ID":"b032a2d6-c0e0-46cf-9265-e214157142f2","Type":"ContainerDied","Data":"4588af9a72c9a03212fef8405c6d480257c966623caa28e39896987d6097e4f3"}
Oct 29 22:45:22.124201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.076109    4860 scope.go:122] "RemoveContainer" containerID="820328f8dfb8251397fc38631bf22a7e7e21d21019db7443d19d5963876a037c"
Oct 29 22:45:22.163786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.163764    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793ee6ac-d14d-480f-93d2-32bba5a4aed5-kube-api-access-7jgc6\" (UniqueName: \"kubernetes.io/projected/793ee6ac-d14d-480f-93d2-32bba5a4aed5-kube-api-access-7jgc6\") pod \"793ee6ac-d14d-480f-93d2-32bba5a4aed5\" (UID: \"793ee6ac-d14d-480f-93d2-32bba5a4aed5\") "
Oct 29 22:45:22.165809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.165778    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793ee6ac-d14d-480f-93d2-32bba5a4aed5-maas-api-service-ca" pod "793ee6ac-d14d-480f-93d2-32bba5a4aed5" (UID: "793ee6ac-d14d-480f-93d2-32bba5a4aed5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.166020 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.166001    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/793ee6ac-d14d-480f-93d2-32bba5a4aed5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/793ee6ac-d14d-480f-93d2-32bba5a4aed5-maas-api-service-ca\") pod \"793ee6ac-d14d-480f-93d2-32bba5a4aed5\" (UID: \"793ee6ac-d14d-480f-93d2-32bba5a4aed5\") "
Oct 29 22:45:22.169294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.169272    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/793ee6ac-d14d-480f-93d2-32bba5a4aed5-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.210566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.210508    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793ee6ac-d14d-480f-93d2-32bba5a4aed5-kube-api-access-7jgc6" pod "793ee6ac-d14d-480f-93d2-32bba5a4aed5" (UID: "793ee6ac-d14d-480f-93d2-32bba5a4aed5"). InnerVolumeSpecName "kube-api-access-7jgc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:22.308353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.306073    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jgc6\" (UniqueName: \"kubernetes.io/projected/793ee6ac-d14d-480f-93d2-32bba5a4aed5-kube-api-access-7jgc6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.403341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.403286    4860 scope.go:122] "RemoveContainer" containerID="8b10bd2f1e8d73fc4b6aa67b792ce5672f8bd307d4b38352ab0cae697da0bbe4"
Oct 29 22:45:22.419560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.419534    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"]
Oct 29 22:45:22.425510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.425446    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c296a3-29888565-t7wsj"]
Oct 29 22:45:22.458212 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:22.458171    4860 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455b3b28_47ae_40e5_a94d_c9ca25dc15c0.slice/crio-98fbd519e05a0485bf26480b1606d898fd311547dc32a6cba7d3f8effedf4830.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793ee6ac_d14d_480f_93d2_32bba5a4aed5.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:45:22.506543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.504748    4860 scope.go:122] "RemoveContainer" containerID="31ee2f10b5234f1e0adab086d831324a113682dc3f0de8eb966b6c22143ca707"
Oct 29 22:45:22.516886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.516513    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"
Oct 29 22:45:22.621992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.613499    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46509b71-e375-40ca-9a04-d35ddaa6e8bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46509b71-e375-40ca-9a04-d35ddaa6e8bc-maas-api-service-ca\") pod \"46509b71-e375-40ca-9a04-d35ddaa6e8bc\" (UID: \"46509b71-e375-40ca-9a04-d35ddaa6e8bc\") "
Oct 29 22:45:22.621992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.613716    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46509b71-e375-40ca-9a04-d35ddaa6e8bc-kube-api-access-5dhbq\" (UniqueName: \"kubernetes.io/projected/46509b71-e375-40ca-9a04-d35ddaa6e8bc-kube-api-access-5dhbq\") pod \"46509b71-e375-40ca-9a04-d35ddaa6e8bc\" (UID: \"46509b71-e375-40ca-9a04-d35ddaa6e8bc\") "
Oct 29 22:45:22.621992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.621949    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46509b71-e375-40ca-9a04-d35ddaa6e8bc-maas-api-service-ca" pod "46509b71-e375-40ca-9a04-d35ddaa6e8bc" (UID: "46509b71-e375-40ca-9a04-d35ddaa6e8bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.651466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.651161    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46509b71-e375-40ca-9a04-d35ddaa6e8bc-kube-api-access-5dhbq" pod "46509b71-e375-40ca-9a04-d35ddaa6e8bc" (UID: "46509b71-e375-40ca-9a04-d35ddaa6e8bc"). InnerVolumeSpecName "kube-api-access-5dhbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:22.706280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.705565    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"
Oct 29 22:45:22.720679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.720651    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/696b181a-c751-4741-8f95-86a613ef965c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696b181a-c751-4741-8f95-86a613ef965c-maas-api-service-ca\") pod \"696b181a-c751-4741-8f95-86a613ef965c\" (UID: \"696b181a-c751-4741-8f95-86a613ef965c\") "
Oct 29 22:45:22.720823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.720754    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/696b181a-c751-4741-8f95-86a613ef965c-kube-api-access-7z8wl\" (UniqueName: \"kubernetes.io/projected/696b181a-c751-4741-8f95-86a613ef965c-kube-api-access-7z8wl\") pod \"696b181a-c751-4741-8f95-86a613ef965c\" (UID: \"696b181a-c751-4741-8f95-86a613ef965c\") "
Oct 29 22:45:22.721693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.721361    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dhbq\" (UniqueName: \"kubernetes.io/projected/46509b71-e375-40ca-9a04-d35ddaa6e8bc-kube-api-access-5dhbq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.721693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.721387    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46509b71-e375-40ca-9a04-d35ddaa6e8bc-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.721926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.721796    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696b181a-c751-4741-8f95-86a613ef965c-maas-api-service-ca" pod "696b181a-c751-4741-8f95-86a613ef965c" (UID: "696b181a-c751-4741-8f95-86a613ef965c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.807222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.806964    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696b181a-c751-4741-8f95-86a613ef965c-kube-api-access-7z8wl" pod "696b181a-c751-4741-8f95-86a613ef965c" (UID: "696b181a-c751-4741-8f95-86a613ef965c"). InnerVolumeSpecName "kube-api-access-7z8wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:22.824128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.824093    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696b181a-c751-4741-8f95-86a613ef965c-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.824128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.824126    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z8wl\" (UniqueName: \"kubernetes.io/projected/696b181a-c751-4741-8f95-86a613ef965c-kube-api-access-7z8wl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:22.863198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.863170    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"
Oct 29 22:45:22.867142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.867122    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"
Oct 29 22:45:22.914727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.914578    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"]
Oct 29 22:45:22.925315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.925279    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2580b08-b5bb-4306-a829-db8b7d832781-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2580b08-b5bb-4306-a829-db8b7d832781-maas-api-service-ca\") pod \"c2580b08-b5bb-4306-a829-db8b7d832781\" (UID: \"c2580b08-b5bb-4306-a829-db8b7d832781\") "
Oct 29 22:45:22.925497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.925398    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2580b08-b5bb-4306-a829-db8b7d832781-kube-api-access-f58gp\" (UniqueName: \"kubernetes.io/projected/c2580b08-b5bb-4306-a829-db8b7d832781-kube-api-access-f58gp\") pod \"c2580b08-b5bb-4306-a829-db8b7d832781\" (UID: \"c2580b08-b5bb-4306-a829-db8b7d832781\") "
Oct 29 22:45:22.925568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.925531    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b032a2d6-c0e0-46cf-9265-e214157142f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b032a2d6-c0e0-46cf-9265-e214157142f2-maas-api-service-ca\") pod \"b032a2d6-c0e0-46cf-9265-e214157142f2\" (UID: \"b032a2d6-c0e0-46cf-9265-e214157142f2\") "
Oct 29 22:45:22.925769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.925745    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b032a2d6-c0e0-46cf-9265-e214157142f2-kube-api-access-s825d\" (UniqueName: \"kubernetes.io/projected/b032a2d6-c0e0-46cf-9265-e214157142f2-kube-api-access-s825d\") pod \"b032a2d6-c0e0-46cf-9265-e214157142f2\" (UID: \"b032a2d6-c0e0-46cf-9265-e214157142f2\") "
Oct 29 22:45:22.927426 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.927344    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b032a2d6-c0e0-46cf-9265-e214157142f2-maas-api-service-ca" pod "b032a2d6-c0e0-46cf-9265-e214157142f2" (UID: "b032a2d6-c0e0-46cf-9265-e214157142f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.927657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.927629    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2580b08-b5bb-4306-a829-db8b7d832781-maas-api-service-ca" pod "c2580b08-b5bb-4306-a829-db8b7d832781" (UID: "c2580b08-b5bb-4306-a829-db8b7d832781"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:22.929597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.929570    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b032a2d6-c0e0-46cf-9265-e214157142f2-kube-api-access-s825d" pod "b032a2d6-c0e0-46cf-9265-e214157142f2" (UID: "b032a2d6-c0e0-46cf-9265-e214157142f2"). InnerVolumeSpecName "kube-api-access-s825d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:22.930250 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:22.930228    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2580b08-b5bb-4306-a829-db8b7d832781-kube-api-access-f58gp" pod "c2580b08-b5bb-4306-a829-db8b7d832781" (UID: "c2580b08-b5bb-4306-a829-db8b7d832781"). InnerVolumeSpecName "kube-api-access-f58gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:23.028805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.028772    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2580b08-b5bb-4306-a829-db8b7d832781-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:23.028805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.028800    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f58gp\" (UniqueName: \"kubernetes.io/projected/c2580b08-b5bb-4306-a829-db8b7d832781-kube-api-access-f58gp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:23.028805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.028809    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b032a2d6-c0e0-46cf-9265-e214157142f2-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:23.029092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.028818    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s825d\" (UniqueName: \"kubernetes.io/projected/b032a2d6-c0e0-46cf-9265-e214157142f2-kube-api-access-s825d\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:23.064138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.064096    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds" event={"ID":"b032a2d6-c0e0-46cf-9265-e214157142f2","Type":"ContainerDied","Data":"20e5058cee851781ad8e851c6cf90a95ef6e5f4b2b19e8c6cbfe01da88cae8e2"}
Oct 29 22:45:23.064311 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.064234    4860 scope.go:122] "RemoveContainer" containerID="4588af9a72c9a03212fef8405c6d480257c966623caa28e39896987d6097e4f3"
Oct 29 22:45:23.064503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.064475    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"
Oct 29 22:45:23.066750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.066726    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc" event={"ID":"f00c15ba-e735-40a2-a1be-f32f20d8d7be","Type":"ContainerStarted","Data":"d2ac60fcc68ad145f20162762f43d5982caed1040d09dd05749df74a31bca7c1"}
Oct 29 22:45:23.085263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.072975    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns" event={"ID":"41a54691-ca33-41d5-bdf2-f3d603727ee3","Type":"ContainerStarted","Data":"86dc3b146ca09e4eea0e3b7f2db073dc25c4487f5f45f1db476d67ae318ea79a"}
Oct 29 22:45:23.085263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.077783    4860 generic.go:356] "Generic (PLEG): container finished" podID="8632936a-6257-4b14-9607-d7b18dfb85f7" containerID="1c8e42d821f197b0b7d4003f3964f474bc063dbfc43eb10544571475227fb7ee" exitCode=60
Oct 29 22:45:23.085263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.077897    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk" event={"ID":"8632936a-6257-4b14-9607-d7b18dfb85f7","Type":"ContainerDied","Data":"1c8e42d821f197b0b7d4003f3964f474bc063dbfc43eb10544571475227fb7ee"}
Oct 29 22:45:23.085263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.082184    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs" event={"ID":"46509b71-e375-40ca-9a04-d35ddaa6e8bc","Type":"ContainerDied","Data":"a2c154991fe8cea6776ddd8c18f1002ed71c63dc55f3c07eb6f242cdb222121e"}
Oct 29 22:45:23.085263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.082282    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"
Oct 29 22:45:23.105847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.098439    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2" event={"ID":"3b8a9038-c7c7-497b-b177-8c8342af29a4","Type":"ContainerStarted","Data":"d3b5333d3410a1bdb10a965f61f625f55374025ff4d1d61592250c7fd99c8dff"}
Oct 29 22:45:23.105847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.101109    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594" event={"ID":"696b181a-c751-4741-8f95-86a613ef965c","Type":"ContainerDied","Data":"de758f9995fff8450fb0dbfd2273869908f6b6da5807a8904d64611c129a6b7b"}
Oct 29 22:45:23.105847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.101201    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"
Oct 29 22:45:23.108580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.108459    4860 generic.go:356] "Generic (PLEG): container finished" podID="b6184068-b2bf-487b-9b92-200dd0b4fd10" containerID="d4bcbcab3aaf1ef0afb2fb7c09b09c149cf4e4eff05a5de91c898898c15e745a" exitCode=60
Oct 29 22:45:23.108580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.108538    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" event={"ID":"b6184068-b2bf-487b-9b92-200dd0b4fd10","Type":"ContainerDied","Data":"d4bcbcab3aaf1ef0afb2fb7c09b09c149cf4e4eff05a5de91c898898c15e745a"}
Oct 29 22:45:23.109570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.109545    4860 scope.go:122] "RemoveContainer" containerID="d4bcbcab3aaf1ef0afb2fb7c09b09c149cf4e4eff05a5de91c898898c15e745a"
Oct 29 22:45:23.110589 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:23.109865    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78_odh-ai-gateway-infra(b6184068-b2bf-487b-9b92-200dd0b4fd10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10"
Oct 29 22:45:23.111595 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.111570    4860 generic.go:356] "Generic (PLEG): container finished" podID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" containerID="98fbd519e05a0485bf26480b1606d898fd311547dc32a6cba7d3f8effedf4830" exitCode=60
Oct 29 22:45:23.111673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.111659    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw" event={"ID":"455b3b28-47ae-40e5-a94d-c9ca25dc15c0","Type":"ContainerDied","Data":"98fbd519e05a0485bf26480b1606d898fd311547dc32a6cba7d3f8effedf4830"}
Oct 29 22:45:23.112081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.112066    4860 scope.go:122] "RemoveContainer" containerID="98fbd519e05a0485bf26480b1606d898fd311547dc32a6cba7d3f8effedf4830"
Oct 29 22:45:23.112415 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:23.112393    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw_odh-ai-gateway-infra(455b3b28-47ae-40e5-a94d-c9ca25dc15c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0"
Oct 29 22:45:23.118756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.118728    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4" event={"ID":"c2580b08-b5bb-4306-a829-db8b7d832781","Type":"ContainerDied","Data":"92d1c99395e13b8d3f200dec32ecbed83e7151272cfb62b0b6d2d277a46d31a4"}
Oct 29 22:45:23.118910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.118894    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"
Oct 29 22:45:23.128002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.127972    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w" event={"ID":"2f97838e-68b7-4252-93dd-f535f4cca7f1","Type":"ContainerStarted","Data":"854d0bbed75cea505e532fef3da0569d7313d178c80adb8e256687a4289fcef0"}
Oct 29 22:45:23.131017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.130990    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"]
Oct 29 22:45:23.160300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.159649    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767" event={"ID":"6d1c0aa4-106c-44cf-8c66-c94717ba5c48","Type":"ContainerStarted","Data":"cf1e5110c797ab781ac3962a089c4356622e2031c45fe4a838f5deeb9db1ca28"}
Oct 29 22:45:23.160300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.160267    4860 scope.go:122] "RemoveContainer" containerID="cf1e5110c797ab781ac3962a089c4356622e2031c45fe4a838f5deeb9db1ca28"
Oct 29 22:45:23.164100 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:23.160882    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767_odh-ai-gateway-infra(6d1c0aa4-106c-44cf-8c66-c94717ba5c48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48"
Oct 29 22:45:23.164100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.163360    4860 scope.go:122] "RemoveContainer" containerID="0157b190ec11bcca313fcae71de4c1a068dd9f21280e3be64d110be794263ed2"
Oct 29 22:45:23.167158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.167130    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng" event={"ID":"793ee6ac-d14d-480f-93d2-32bba5a4aed5","Type":"ContainerDied","Data":"4720a73507db5ad8eb1037de1a090087677d68d2e1bfa1aff7cf38fbbf34d1d6"}
Oct 29 22:45:23.167280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.167257    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"
Oct 29 22:45:23.254855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.254832    4860 scope.go:122] "RemoveContainer" containerID="5ebae55fbd731c61488c40d3cf6d415ea0d778d0a8ca9e86dee7e8d0584ad0d9"
Oct 29 22:45:23.299359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.293713    4860 scope.go:122] "RemoveContainer" containerID="54998d1e3333404692db7f3339f71955bd8e8723c16f4cb9230ea5f65582b1cd"
Oct 29 22:45:23.311055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.311009    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:23.421431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.420112    4860 scope.go:122] "RemoveContainer" containerID="c2788439c0acf87217aae0960227913dc5ddea49bf07dd5c443944f083dbc0fe"
Oct 29 22:45:23.432170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.432140    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"
Oct 29 22:45:23.442583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.442554    4860 scope.go:122] "RemoveContainer" containerID="09f31566232b809306f4cdd8cf6b660b816c79a97de02546683838e9f0b377a2"
Oct 29 22:45:23.451918 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.451875    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w" podStartSLOduration=23.451861137 podStartE2EDuration="23.451861137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:23.44970833 +0000 UTC m=+1865.474484481" watchObservedRunningTime="2026-10-29 22:45:23.451861137 +0000 UTC m=+1865.476637267"
Oct 29 22:45:23.460017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.459990    4860 scope.go:122] "RemoveContainer" containerID="14f14a79d27a92db301a45afc0aaa21c71b4963e37b96b44b97d0b44c3b6348e"
Oct 29 22:45:23.523440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.523416    4860 scope.go:122] "RemoveContainer" containerID="d7b4d5b12f5aef941e65e00a9c098e64427cb32a7ded96b29e4e581532b9424e"
Oct 29 22:45:23.528996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.528967    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"]
Oct 29 22:45:23.537529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.537506    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a5af18-29888565-8tgng"]
Oct 29 22:45:23.551140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.551114    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8632936a-6257-4b14-9607-d7b18dfb85f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8632936a-6257-4b14-9607-d7b18dfb85f7-maas-api-service-ca\") pod \"8632936a-6257-4b14-9607-d7b18dfb85f7\" (UID: \"8632936a-6257-4b14-9607-d7b18dfb85f7\") "
Oct 29 22:45:23.551242 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.551171    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8632936a-6257-4b14-9607-d7b18dfb85f7-kube-api-access-m2b9h\" (UniqueName: \"kubernetes.io/projected/8632936a-6257-4b14-9607-d7b18dfb85f7-kube-api-access-m2b9h\") pod \"8632936a-6257-4b14-9607-d7b18dfb85f7\" (UID: \"8632936a-6257-4b14-9607-d7b18dfb85f7\") "
Oct 29 22:45:23.551508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.551484    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8632936a-6257-4b14-9607-d7b18dfb85f7-maas-api-service-ca" pod "8632936a-6257-4b14-9607-d7b18dfb85f7" (UID: "8632936a-6257-4b14-9607-d7b18dfb85f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:23.552465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.552441    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8632936a-6257-4b14-9607-d7b18dfb85f7-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:23.553302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.553280    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8632936a-6257-4b14-9607-d7b18dfb85f7-kube-api-access-m2b9h" pod "8632936a-6257-4b14-9607-d7b18dfb85f7" (UID: "8632936a-6257-4b14-9607-d7b18dfb85f7"). InnerVolumeSpecName "kube-api-access-m2b9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:23.563802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.563780    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"]
Oct 29 22:45:23.626356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.626314    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2e046a-29888565-xmgds"]
Oct 29 22:45:23.654538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.654500    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2b9h\" (UniqueName: \"kubernetes.io/projected/8632936a-6257-4b14-9607-d7b18dfb85f7-kube-api-access-m2b9h\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:23.735515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.735477    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"]
Oct 29 22:45:23.741872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.741841    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c1750b-29888565-gj594"]
Oct 29 22:45:23.799197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.799164    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"]
Oct 29 22:45:23.808193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.808155    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-08ebee-29888565-rrld4"]
Oct 29 22:45:23.832864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.832832    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"]
Oct 29 22:45:23.837962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:23.837936    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c7d192-29888565-qthrs"]
Oct 29 22:45:24.118240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.118161    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" path="/var/lib/kubelet/pods/1276947c-a4b8-4eb3-8aaf-2d589e81b84c/volumes"
Oct 29 22:45:24.118764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.118747    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" path="/var/lib/kubelet/pods/46509b71-e375-40ca-9a04-d35ddaa6e8bc/volumes"
Oct 29 22:45:24.119279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.119263    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="696b181a-c751-4741-8f95-86a613ef965c" path="/var/lib/kubelet/pods/696b181a-c751-4741-8f95-86a613ef965c/volumes"
Oct 29 22:45:24.120073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.120055    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" path="/var/lib/kubelet/pods/793ee6ac-d14d-480f-93d2-32bba5a4aed5/volumes"
Oct 29 22:45:24.120530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.120515    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b032a2d6-c0e0-46cf-9265-e214157142f2" path="/var/lib/kubelet/pods/b032a2d6-c0e0-46cf-9265-e214157142f2/volumes"
Oct 29 22:45:24.120947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.120932    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2580b08-b5bb-4306-a829-db8b7d832781" path="/var/lib/kubelet/pods/c2580b08-b5bb-4306-a829-db8b7d832781/volumes"
Oct 29 22:45:24.207271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.207231    4860 generic.go:356] "Generic (PLEG): container finished" podID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" containerID="d2ac60fcc68ad145f20162762f43d5982caed1040d09dd05749df74a31bca7c1" exitCode=60
Oct 29 22:45:24.207271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.207255    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc" event={"ID":"f00c15ba-e735-40a2-a1be-f32f20d8d7be","Type":"ContainerDied","Data":"d2ac60fcc68ad145f20162762f43d5982caed1040d09dd05749df74a31bca7c1"}
Oct 29 22:45:24.207572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.207319    4860 scope.go:122] "RemoveContainer" containerID="d90b5955ea1d29ca48e385fb047b2ae81ec30c3ce9536d5d86719c87fab07979"
Oct 29 22:45:24.207889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.207867    4860 scope.go:122] "RemoveContainer" containerID="d2ac60fcc68ad145f20162762f43d5982caed1040d09dd05749df74a31bca7c1"
Oct 29 22:45:24.208498 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:24.208273    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc_odh-ai-gateway-infra(f00c15ba-e735-40a2-a1be-f32f20d8d7be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be"
Oct 29 22:45:24.209841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.209819    4860 generic.go:356] "Generic (PLEG): container finished" podID="41a54691-ca33-41d5-bdf2-f3d603727ee3" containerID="86dc3b146ca09e4eea0e3b7f2db073dc25c4487f5f45f1db476d67ae318ea79a" exitCode=60
Oct 29 22:45:24.209947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.209893    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns" event={"ID":"41a54691-ca33-41d5-bdf2-f3d603727ee3","Type":"ContainerDied","Data":"86dc3b146ca09e4eea0e3b7f2db073dc25c4487f5f45f1db476d67ae318ea79a"}
Oct 29 22:45:24.210320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.210303    4860 scope.go:122] "RemoveContainer" containerID="86dc3b146ca09e4eea0e3b7f2db073dc25c4487f5f45f1db476d67ae318ea79a"
Oct 29 22:45:24.210725 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:24.210703    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns_odh-ai-gateway-infra(41a54691-ca33-41d5-bdf2-f3d603727ee3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3"
Oct 29 22:45:24.211707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.211505    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"
Oct 29 22:45:24.211707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.211546    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk" event={"ID":"8632936a-6257-4b14-9607-d7b18dfb85f7","Type":"ContainerDied","Data":"86b2d8855ac0263075c4d3be849dc46a6be2d818434d7703fac1a75c10e0a10e"}
Oct 29 22:45:24.215294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.215272    4860 generic.go:356] "Generic (PLEG): container finished" podID="3b8a9038-c7c7-497b-b177-8c8342af29a4" containerID="d3b5333d3410a1bdb10a965f61f625f55374025ff4d1d61592250c7fd99c8dff" exitCode=60
Oct 29 22:45:24.215426 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.215407    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2" event={"ID":"3b8a9038-c7c7-497b-b177-8c8342af29a4","Type":"ContainerDied","Data":"d3b5333d3410a1bdb10a965f61f625f55374025ff4d1d61592250c7fd99c8dff"}
Oct 29 22:45:24.215776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.215756    4860 scope.go:122] "RemoveContainer" containerID="d3b5333d3410a1bdb10a965f61f625f55374025ff4d1d61592250c7fd99c8dff"
Oct 29 22:45:24.216079 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:24.216055    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2_odh-ai-gateway-infra(3b8a9038-c7c7-497b-b177-8c8342af29a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4"
Oct 29 22:45:24.224721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.224701    4860 generic.go:356] "Generic (PLEG): container finished" podID="2f97838e-68b7-4252-93dd-f535f4cca7f1" containerID="854d0bbed75cea505e532fef3da0569d7313d178c80adb8e256687a4289fcef0" exitCode=60
Oct 29 22:45:24.224827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.224779    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w" event={"ID":"2f97838e-68b7-4252-93dd-f535f4cca7f1","Type":"ContainerDied","Data":"854d0bbed75cea505e532fef3da0569d7313d178c80adb8e256687a4289fcef0"}
Oct 29 22:45:24.225180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.225157    4860 scope.go:122] "RemoveContainer" containerID="854d0bbed75cea505e532fef3da0569d7313d178c80adb8e256687a4289fcef0"
Oct 29 22:45:24.227368 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:24.225461    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w_odh-ai-gateway-infra(2f97838e-68b7-4252-93dd-f535f4cca7f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1"
Oct 29 22:45:24.227368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.226390    4860 scope.go:122] "RemoveContainer" containerID="d7a5884366c9cb4517b759fe6bd358ea13ccfcd8783de119839e1dfb6115b48f"
Oct 29 22:45:24.228949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.228143    4860 generic.go:356] "Generic (PLEG): container finished" podID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" containerID="cf1e5110c797ab781ac3962a089c4356622e2031c45fe4a838f5deeb9db1ca28" exitCode=60
Oct 29 22:45:24.228949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.228204    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767" event={"ID":"6d1c0aa4-106c-44cf-8c66-c94717ba5c48","Type":"ContainerDied","Data":"cf1e5110c797ab781ac3962a089c4356622e2031c45fe4a838f5deeb9db1ca28"}
Oct 29 22:45:24.228949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.228588    4860 scope.go:122] "RemoveContainer" containerID="cf1e5110c797ab781ac3962a089c4356622e2031c45fe4a838f5deeb9db1ca28"
Oct 29 22:45:24.228949 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:24.228886    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767_odh-ai-gateway-infra(6d1c0aa4-106c-44cf-8c66-c94717ba5c48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48"
Oct 29 22:45:24.246314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.246291    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"]
Oct 29 22:45:24.248995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.248977    4860 scope.go:122] "RemoveContainer" containerID="1c8e42d821f197b0b7d4003f3964f474bc063dbfc43eb10544571475227fb7ee"
Oct 29 22:45:24.263506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.263248    4860 scope.go:122] "RemoveContainer" containerID="d8bc58ff8db7cb0e98df1537328f8c9d620ccc90e345dc2648b92638a3f5bf9f"
Oct 29 22:45:24.317172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.317141    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"]
Oct 29 22:45:24.318855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.318567    4860 scope.go:122] "RemoveContainer" containerID="43406f9b774268782c41aeda31f4e7d2544ba63d1ba7cde8fd655c22a84f7b86"
Oct 29 22:45:24.344751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.344727    4860 scope.go:122] "RemoveContainer" containerID="2a894177ef66108bb48c355543d6eb51444cee713d3d0ea8cadcef6150ee2d28"
Oct 29 22:45:24.412764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.412729    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:24.429131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.429099    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"]
Oct 29 22:45:24.513062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.513028    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"]
Oct 29 22:45:24.815380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.796387    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"]
Oct 29 22:45:24.824349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.824306    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2a3ef0-29888565-wljxk"]
Oct 29 22:45:24.828011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.827154    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"
Oct 29 22:45:24.830852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.830829    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"]
Oct 29 22:45:24.914290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.914258    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"]
Oct 29 22:45:24.966412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.963941    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6184068-b2bf-487b-9b92-200dd0b4fd10-kube-api-access-r9gfr\" (UniqueName: \"kubernetes.io/projected/b6184068-b2bf-487b-9b92-200dd0b4fd10-kube-api-access-r9gfr\") pod \"b6184068-b2bf-487b-9b92-200dd0b4fd10\" (UID: \"b6184068-b2bf-487b-9b92-200dd0b4fd10\") "
Oct 29 22:45:24.966412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.964232    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6184068-b2bf-487b-9b92-200dd0b4fd10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6184068-b2bf-487b-9b92-200dd0b4fd10-maas-api-service-ca\") pod \"b6184068-b2bf-487b-9b92-200dd0b4fd10\" (UID: \"b6184068-b2bf-487b-9b92-200dd0b4fd10\") "
Oct 29 22:45:24.966412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.965398    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6184068-b2bf-487b-9b92-200dd0b4fd10-maas-api-service-ca" pod "b6184068-b2bf-487b-9b92-200dd0b4fd10" (UID: "b6184068-b2bf-487b-9b92-200dd0b4fd10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:24.993489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:24.993456    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6184068-b2bf-487b-9b92-200dd0b4fd10-kube-api-access-r9gfr" pod "b6184068-b2bf-487b-9b92-200dd0b4fd10" (UID: "b6184068-b2bf-487b-9b92-200dd0b4fd10"). InnerVolumeSpecName "kube-api-access-r9gfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:25.072354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.067306    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6184068-b2bf-487b-9b92-200dd0b4fd10-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:25.072643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.072621    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9gfr\" (UniqueName: \"kubernetes.io/projected/b6184068-b2bf-487b-9b92-200dd0b4fd10-kube-api-access-r9gfr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:25.161510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.161472    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"
Oct 29 22:45:25.251812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.251782    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78" event={"ID":"b6184068-b2bf-487b-9b92-200dd0b4fd10","Type":"ContainerDied","Data":"3eaf85017f2bec319a8a9295f1598acd4afa5292198a1fccae203bc65cf5d0fe"}
Oct 29 22:45:25.251939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.251846    4860 scope.go:122] "RemoveContainer" containerID="d4bcbcab3aaf1ef0afb2fb7c09b09c149cf4e4eff05a5de91c898898c15e745a"
Oct 29 22:45:25.252013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.251939    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"
Oct 29 22:45:25.263693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.263662    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw" event={"ID":"455b3b28-47ae-40e5-a94d-c9ca25dc15c0","Type":"ContainerDied","Data":"fbfa8f4798f2ab32a80efac5246b366e7568c4bf7e5a63d5b4e1b0e4a63511bc"}
Oct 29 22:45:25.263824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.263781    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"
Oct 29 22:45:25.299423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.299382    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-kube-api-access-27rzc\" (UniqueName: \"kubernetes.io/projected/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-kube-api-access-27rzc\") pod \"455b3b28-47ae-40e5-a94d-c9ca25dc15c0\" (UID: \"455b3b28-47ae-40e5-a94d-c9ca25dc15c0\") "
Oct 29 22:45:25.299649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.299626    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-maas-api-service-ca\") pod \"455b3b28-47ae-40e5-a94d-c9ca25dc15c0\" (UID: \"455b3b28-47ae-40e5-a94d-c9ca25dc15c0\") "
Oct 29 22:45:25.300988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.300771    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-maas-api-service-ca" pod "455b3b28-47ae-40e5-a94d-c9ca25dc15c0" (UID: "455b3b28-47ae-40e5-a94d-c9ca25dc15c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:25.308165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.306717    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-kube-api-access-27rzc" pod "455b3b28-47ae-40e5-a94d-c9ca25dc15c0" (UID: "455b3b28-47ae-40e5-a94d-c9ca25dc15c0"). InnerVolumeSpecName "kube-api-access-27rzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:25.324183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.322440    4860 scope.go:122] "RemoveContainer" containerID="98fbd519e05a0485bf26480b1606d898fd311547dc32a6cba7d3f8effedf4830"
Oct 29 22:45:25.328466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.327822    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:25.427345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.425627    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rzc\" (UniqueName: \"kubernetes.io/projected/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-kube-api-access-27rzc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:25.427345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.425670    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/455b3b28-47ae-40e5-a94d-c9ca25dc15c0-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:25.461086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.458827    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"]
Oct 29 22:45:25.461086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.460893    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8147a8-29888565-2lr78"]
Oct 29 22:45:25.778263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.778203    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"]
Oct 29 22:45:25.785605 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.785575    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fc684b-29888565-bk6fw"]
Oct 29 22:45:25.787434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.787412    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"
Oct 29 22:45:25.838572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.837366    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8a9038-c7c7-497b-b177-8c8342af29a4-kube-api-access-brrfk\" (UniqueName: \"kubernetes.io/projected/3b8a9038-c7c7-497b-b177-8c8342af29a4-kube-api-access-brrfk\") pod \"3b8a9038-c7c7-497b-b177-8c8342af29a4\" (UID: \"3b8a9038-c7c7-497b-b177-8c8342af29a4\") "
Oct 29 22:45:25.838572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.837528    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8a9038-c7c7-497b-b177-8c8342af29a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8a9038-c7c7-497b-b177-8c8342af29a4-maas-api-service-ca\") pod \"3b8a9038-c7c7-497b-b177-8c8342af29a4\" (UID: \"3b8a9038-c7c7-497b-b177-8c8342af29a4\") "
Oct 29 22:45:25.838926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.838889    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8a9038-c7c7-497b-b177-8c8342af29a4-maas-api-service-ca" pod "3b8a9038-c7c7-497b-b177-8c8342af29a4" (UID: "3b8a9038-c7c7-497b-b177-8c8342af29a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:25.842692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.842666    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8a9038-c7c7-497b-b177-8c8342af29a4-kube-api-access-brrfk" pod "3b8a9038-c7c7-497b-b177-8c8342af29a4" (UID: "3b8a9038-c7c7-497b-b177-8c8342af29a4"). InnerVolumeSpecName "kube-api-access-brrfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:25.873914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.870427    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"
Oct 29 22:45:25.875009 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.874983    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"
Oct 29 22:45:25.881583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.881555    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"
Oct 29 22:45:25.884467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.884448    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"
Oct 29 22:45:25.939582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.938879    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f97838e-68b7-4252-93dd-f535f4cca7f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f97838e-68b7-4252-93dd-f535f4cca7f1-maas-api-service-ca\") pod \"2f97838e-68b7-4252-93dd-f535f4cca7f1\" (UID: \"2f97838e-68b7-4252-93dd-f535f4cca7f1\") "
Oct 29 22:45:25.939582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.939340    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a54691-ca33-41d5-bdf2-f3d603727ee3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a54691-ca33-41d5-bdf2-f3d603727ee3-maas-api-service-ca\") pod \"41a54691-ca33-41d5-bdf2-f3d603727ee3\" (UID: \"41a54691-ca33-41d5-bdf2-f3d603727ee3\") "
Oct 29 22:45:25.939582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.939500    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f00c15ba-e735-40a2-a1be-f32f20d8d7be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00c15ba-e735-40a2-a1be-f32f20d8d7be-maas-api-service-ca\") pod \"f00c15ba-e735-40a2-a1be-f32f20d8d7be\" (UID: \"f00c15ba-e735-40a2-a1be-f32f20d8d7be\") "
Oct 29 22:45:25.939582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.939548    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-kube-api-access-2spwl\" (UniqueName: \"kubernetes.io/projected/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-kube-api-access-2spwl\") pod \"6d1c0aa4-106c-44cf-8c66-c94717ba5c48\" (UID: \"6d1c0aa4-106c-44cf-8c66-c94717ba5c48\") "
Oct 29 22:45:25.939896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.939656    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f97838e-68b7-4252-93dd-f535f4cca7f1-kube-api-access-2dswm\" (UniqueName: \"kubernetes.io/projected/2f97838e-68b7-4252-93dd-f535f4cca7f1-kube-api-access-2dswm\") pod \"2f97838e-68b7-4252-93dd-f535f4cca7f1\" (UID: \"2f97838e-68b7-4252-93dd-f535f4cca7f1\") "
Oct 29 22:45:25.939896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.939742    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00c15ba-e735-40a2-a1be-f32f20d8d7be-kube-api-access-qmlxz\" (UniqueName: \"kubernetes.io/projected/f00c15ba-e735-40a2-a1be-f32f20d8d7be-kube-api-access-qmlxz\") pod \"f00c15ba-e735-40a2-a1be-f32f20d8d7be\" (UID: \"f00c15ba-e735-40a2-a1be-f32f20d8d7be\") "
Oct 29 22:45:25.939896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.939869    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-maas-api-service-ca\") pod \"6d1c0aa4-106c-44cf-8c66-c94717ba5c48\" (UID: \"6d1c0aa4-106c-44cf-8c66-c94717ba5c48\") "
Oct 29 22:45:25.940067 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.939944    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a54691-ca33-41d5-bdf2-f3d603727ee3-kube-api-access-gp6n7\" (UniqueName: \"kubernetes.io/projected/41a54691-ca33-41d5-bdf2-f3d603727ee3-kube-api-access-gp6n7\") pod \"41a54691-ca33-41d5-bdf2-f3d603727ee3\" (UID: \"41a54691-ca33-41d5-bdf2-f3d603727ee3\") "
Oct 29 22:45:25.940844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.939262    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f97838e-68b7-4252-93dd-f535f4cca7f1-maas-api-service-ca" pod "2f97838e-68b7-4252-93dd-f535f4cca7f1" (UID: "2f97838e-68b7-4252-93dd-f535f4cca7f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:25.940844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.940732    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a54691-ca33-41d5-bdf2-f3d603727ee3-maas-api-service-ca" pod "41a54691-ca33-41d5-bdf2-f3d603727ee3" (UID: "41a54691-ca33-41d5-bdf2-f3d603727ee3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:25.941282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.941121    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-maas-api-service-ca" pod "6d1c0aa4-106c-44cf-8c66-c94717ba5c48" (UID: "6d1c0aa4-106c-44cf-8c66-c94717ba5c48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:25.941835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.941813    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00c15ba-e735-40a2-a1be-f32f20d8d7be-maas-api-service-ca" pod "f00c15ba-e735-40a2-a1be-f32f20d8d7be" (UID: "f00c15ba-e735-40a2-a1be-f32f20d8d7be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:25.942037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.942011    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f97838e-68b7-4252-93dd-f535f4cca7f1-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:25.942120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.942043    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a54691-ca33-41d5-bdf2-f3d603727ee3-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:25.942120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.942058    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8a9038-c7c7-497b-b177-8c8342af29a4-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:25.942120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.942072    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brrfk\" (UniqueName: \"kubernetes.io/projected/3b8a9038-c7c7-497b-b177-8c8342af29a4-kube-api-access-brrfk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:25.942120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.942087    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:25.944382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.944283    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-kube-api-access-2spwl" pod "6d1c0aa4-106c-44cf-8c66-c94717ba5c48" (UID: "6d1c0aa4-106c-44cf-8c66-c94717ba5c48"). InnerVolumeSpecName "kube-api-access-2spwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:25.945666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.945637    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f97838e-68b7-4252-93dd-f535f4cca7f1-kube-api-access-2dswm" pod "2f97838e-68b7-4252-93dd-f535f4cca7f1" (UID: "2f97838e-68b7-4252-93dd-f535f4cca7f1"). InnerVolumeSpecName "kube-api-access-2dswm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:25.946091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.946066    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00c15ba-e735-40a2-a1be-f32f20d8d7be-kube-api-access-qmlxz" pod "f00c15ba-e735-40a2-a1be-f32f20d8d7be" (UID: "f00c15ba-e735-40a2-a1be-f32f20d8d7be"). InnerVolumeSpecName "kube-api-access-qmlxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:25.946827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:25.946801    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a54691-ca33-41d5-bdf2-f3d603727ee3-kube-api-access-gp6n7" pod "41a54691-ca33-41d5-bdf2-f3d603727ee3" (UID: "41a54691-ca33-41d5-bdf2-f3d603727ee3"). InnerVolumeSpecName "kube-api-access-gp6n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:26.044343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.044297    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp6n7\" (UniqueName: \"kubernetes.io/projected/41a54691-ca33-41d5-bdf2-f3d603727ee3-kube-api-access-gp6n7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.044476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.044345    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00c15ba-e735-40a2-a1be-f32f20d8d7be-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.044476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.044364    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2spwl\" (UniqueName: \"kubernetes.io/projected/6d1c0aa4-106c-44cf-8c66-c94717ba5c48-kube-api-access-2spwl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.044476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.044380    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dswm\" (UniqueName: \"kubernetes.io/projected/2f97838e-68b7-4252-93dd-f535f4cca7f1-kube-api-access-2dswm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.044476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.044396    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmlxz\" (UniqueName: \"kubernetes.io/projected/f00c15ba-e735-40a2-a1be-f32f20d8d7be-kube-api-access-qmlxz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.119654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.119620    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" path="/var/lib/kubelet/pods/455b3b28-47ae-40e5-a94d-c9ca25dc15c0/volumes"
Oct 29 22:45:26.120633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.120610    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8632936a-6257-4b14-9607-d7b18dfb85f7" path="/var/lib/kubelet/pods/8632936a-6257-4b14-9607-d7b18dfb85f7/volumes"
Oct 29 22:45:26.121429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.121410    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10" path="/var/lib/kubelet/pods/b6184068-b2bf-487b-9b92-200dd0b4fd10/volumes"
Oct 29 22:45:26.243414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.243388    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:45:26.247275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.247247    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-podinfo\") pod \"e292294d-64b9-43b7-a0b9-89f63da16cdb\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") "
Oct 29 22:45:26.247441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.247291    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-envoy\") pod \"e292294d-64b9-43b7-a0b9-89f63da16cdb\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") "
Oct 29 22:45:26.247441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.247401    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-data\") pod \"e292294d-64b9-43b7-a0b9-89f63da16cdb\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") "
Oct 29 22:45:26.247441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.247424    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-kube-api-access-b8f8h\" (UniqueName: \"kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-kube-api-access-b8f8h\") pod \"e292294d-64b9-43b7-a0b9-89f63da16cdb\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") "
Oct 29 22:45:26.247616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.247472    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-token\" (UniqueName: \"kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-token\") pod \"e292294d-64b9-43b7-a0b9-89f63da16cdb\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") "
Oct 29 22:45:26.247616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.247547    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-certs\") pod \"e292294d-64b9-43b7-a0b9-89f63da16cdb\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") "
Oct 29 22:45:26.247731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.247703    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-credential-socket\") pod \"e292294d-64b9-43b7-a0b9-89f63da16cdb\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") "
Oct 29 22:45:26.247790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.247731    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e292294d-64b9-43b7-a0b9-89f63da16cdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e292294d-64b9-43b7-a0b9-89f63da16cdb-istiod-ca-cert\") pod \"e292294d-64b9-43b7-a0b9-89f63da16cdb\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") "
Oct 29 22:45:26.247848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.247788    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-socket\") pod \"e292294d-64b9-43b7-a0b9-89f63da16cdb\" (UID: \"e292294d-64b9-43b7-a0b9-89f63da16cdb\") "
Oct 29 22:45:26.248206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.248177    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-certs" pod "e292294d-64b9-43b7-a0b9-89f63da16cdb" (UID: "e292294d-64b9-43b7-a0b9-89f63da16cdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:26.248513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.248489    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-credential-socket" pod "e292294d-64b9-43b7-a0b9-89f63da16cdb" (UID: "e292294d-64b9-43b7-a0b9-89f63da16cdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:26.248624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.248603    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.248690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.248631    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.248811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.248788    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e292294d-64b9-43b7-a0b9-89f63da16cdb-istiod-ca-cert" pod "e292294d-64b9-43b7-a0b9-89f63da16cdb" (UID: "e292294d-64b9-43b7-a0b9-89f63da16cdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:26.248883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.248824    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-socket" pod "e292294d-64b9-43b7-a0b9-89f63da16cdb" (UID: "e292294d-64b9-43b7-a0b9-89f63da16cdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:26.249023 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.249005    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-data" pod "e292294d-64b9-43b7-a0b9-89f63da16cdb" (UID: "e292294d-64b9-43b7-a0b9-89f63da16cdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:26.252658 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.252627    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-kube-api-access-b8f8h" pod "e292294d-64b9-43b7-a0b9-89f63da16cdb" (UID: "e292294d-64b9-43b7-a0b9-89f63da16cdb"). InnerVolumeSpecName "kube-api-access-b8f8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:26.253087 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.253061    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-token" pod "e292294d-64b9-43b7-a0b9-89f63da16cdb" (UID: "e292294d-64b9-43b7-a0b9-89f63da16cdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:26.253636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.253611    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-envoy" pod "e292294d-64b9-43b7-a0b9-89f63da16cdb" (UID: "e292294d-64b9-43b7-a0b9-89f63da16cdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:45:26.255224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.255201    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-podinfo" pod "e292294d-64b9-43b7-a0b9-89f63da16cdb" (UID: "e292294d-64b9-43b7-a0b9-89f63da16cdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:45:26.346664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.346549    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"
Oct 29 22:45:26.346664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.346576    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2" event={"ID":"3b8a9038-c7c7-497b-b177-8c8342af29a4","Type":"ContainerDied","Data":"b01c9296a6f4e23d8664ea886a2999fd7bcca7f014893b6f50df76e3a5869e07"}
Oct 29 22:45:26.346664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.346644    4860 scope.go:122] "RemoveContainer" containerID="d3b5333d3410a1bdb10a965f61f625f55374025ff4d1d61592250c7fd99c8dff"
Oct 29 22:45:26.349793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.349768    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e292294d-64b9-43b7-a0b9-89f63da16cdb-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.349793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.349793    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.349959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.349807    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.349959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.349820    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.349959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.349832    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.349959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.349845    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8f8h\" (UniqueName: \"kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-kube-api-access-b8f8h\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.349959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.349858    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e292294d-64b9-43b7-a0b9-89f63da16cdb-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:26.351383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.351360    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w" event={"ID":"2f97838e-68b7-4252-93dd-f535f4cca7f1","Type":"ContainerDied","Data":"fdc3c469ea08ece84b1d795f6536a278e5404e501dba271d7ba3c2be5c665ade"}
Oct 29 22:45:26.351493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.351370    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"
Oct 29 22:45:26.353833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.353675    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767" event={"ID":"6d1c0aa4-106c-44cf-8c66-c94717ba5c48","Type":"ContainerDied","Data":"3cda374cf88852b3ec75f40a7b0b1fa0c4597cf8e381c955c6468e1037258a6d"}
Oct 29 22:45:26.353935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.353921    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"
Oct 29 22:45:26.356543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.356511    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc" event={"ID":"f00c15ba-e735-40a2-a1be-f32f20d8d7be","Type":"ContainerDied","Data":"19167d5487b9e03d501b1517619c1dc3f0818f81e1312a93f1952914c07ace24"}
Oct 29 22:45:26.356705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.356606    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"
Oct 29 22:45:26.359281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.359254    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns" event={"ID":"41a54691-ca33-41d5-bdf2-f3d603727ee3","Type":"ContainerDied","Data":"a6c0513bcc6fcb9c9488c801b727efe559c84faf611813954ceb192b09c6a72d"}
Oct 29 22:45:26.359704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.359683    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"
Oct 29 22:45:26.361172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.361143    4860 generic.go:356] "Generic (PLEG): container finished" podID="e292294d-64b9-43b7-a0b9-89f63da16cdb" containerID="b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64" exitCode=0
Oct 29 22:45:26.361340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.361197    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" event={"ID":"e292294d-64b9-43b7-a0b9-89f63da16cdb","Type":"ContainerDied","Data":"b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64"}
Oct 29 22:45:26.361340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.361205    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"
Oct 29 22:45:26.361340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.361232    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv" event={"ID":"e292294d-64b9-43b7-a0b9-89f63da16cdb","Type":"ContainerDied","Data":"9f609eecc763d08d60e0d6580b0f6248d2916aca4f697aaaeea86255e195b0ba"}
Oct 29 22:45:26.363524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.363500    4860 scope.go:122] "RemoveContainer" containerID="854d0bbed75cea505e532fef3da0569d7313d178c80adb8e256687a4289fcef0"
Oct 29 22:45:26.374686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.374665    4860 scope.go:122] "RemoveContainer" containerID="cf1e5110c797ab781ac3962a089c4356622e2031c45fe4a838f5deeb9db1ca28"
Oct 29 22:45:26.384864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.384846    4860 scope.go:122] "RemoveContainer" containerID="d2ac60fcc68ad145f20162762f43d5982caed1040d09dd05749df74a31bca7c1"
Oct 29 22:45:26.397753 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.397731    4860 scope.go:122] "RemoveContainer" containerID="86dc3b146ca09e4eea0e3b7f2db073dc25c4487f5f45f1db476d67ae318ea79a"
Oct 29 22:45:26.411642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.411622    4860 scope.go:122] "RemoveContainer" containerID="b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64"
Oct 29 22:45:26.413741 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.413720    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"]
Oct 29 22:45:26.420452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.420430    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2ba10a-29888565-77767"]
Oct 29 22:45:26.423756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.423736    4860 scope.go:122] "RemoveContainer" containerID="b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64"
Oct 29 22:45:26.424061 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:26.424041    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64\": container with ID starting with b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64 not found: ID does not exist" containerID="b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64"
Oct 29 22:45:26.424121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.424069    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64"} err="failed to get container status \"b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64\": rpc error: code = NotFound desc = could not find container \"b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64\": container with ID starting with b3e392246126dd9f913fa8e6eb3707f7bcafc762233db4b1a956f264e028bb64 not found: ID does not exist"
Oct 29 22:45:26.466638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.466613    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"]
Oct 29 22:45:26.471643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.471605    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5a69c9-29888565-trkxc"]
Oct 29 22:45:26.508528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.508501    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"]
Oct 29 22:45:26.516494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.516472    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fee294-29888565-m8hns"]
Oct 29 22:45:26.546764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.546742    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"]
Oct 29 22:45:26.551834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.551813    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-db7419-29888565-24ll2"]
Oct 29 22:45:26.587261 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.587235    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"]
Oct 29 22:45:26.592155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.592134    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888565-hvs2w"]
Oct 29 22:45:26.632129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.632046    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"]
Oct 29 22:45:26.637087 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:26.637062    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4d3168-openshift-default-96c48bc6c-4fcgv"]
Oct 29 22:45:27.044695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.044660    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:27.576503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.576463    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:27.715173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.715139    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:27.856747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.856666    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"]
Oct 29 22:45:27.857601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857582    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10" containerName="cleanup"
Oct 29 22:45:27.857661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857603    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10" containerName="cleanup"
Oct 29 22:45:27.857661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857617    4860 cpu_manager.go:414] "containerMap: removing container" podUID="696b181a-c751-4741-8f95-86a613ef965c" containerName="cleanup"
Oct 29 22:45:27.857661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857622    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696b181a-c751-4741-8f95-86a613ef965c" containerName="cleanup"
Oct 29 22:45:27.857661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857631    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" containerName="cleanup"
Oct 29 22:45:27.857661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857636    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" containerName="cleanup"
Oct 29 22:45:27.857661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857642    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c2580b08-b5bb-4306-a829-db8b7d832781" containerName="cleanup"
Oct 29 22:45:27.857661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857647    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2580b08-b5bb-4306-a829-db8b7d832781" containerName="cleanup"
Oct 29 22:45:27.857661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857653    4860 cpu_manager.go:414] "containerMap: removing container" podUID="80f7cc19-3ccf-4e4a-8d30-8f76251885b0" containerName="cleanup"
Oct 29 22:45:27.857661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857657    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f7cc19-3ccf-4e4a-8d30-8f76251885b0" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857667    4860 cpu_manager.go:414] "containerMap: removing container" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857672    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857682    4860 cpu_manager.go:414] "containerMap: removing container" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857688    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857694    4860 cpu_manager.go:414] "containerMap: removing container" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857699    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857705    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857709    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857721    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857725    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857735    4860 cpu_manager.go:414] "containerMap: removing container" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857739    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857746    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8632936a-6257-4b14-9607-d7b18dfb85f7" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857751    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8632936a-6257-4b14-9607-d7b18dfb85f7" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857761    4860 cpu_manager.go:414] "containerMap: removing container" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857765    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857773    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b032a2d6-c0e0-46cf-9265-e214157142f2" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857777    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b032a2d6-c0e0-46cf-9265-e214157142f2" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857783    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857788    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857794    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c2580b08-b5bb-4306-a829-db8b7d832781" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857799    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2580b08-b5bb-4306-a829-db8b7d832781" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857804    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857808    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857815    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c5f2142b-4344-47d4-bdf0-40a45e61bb3e" containerName="maas-api"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857819    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f2142b-4344-47d4-bdf0-40a45e61bb3e" containerName="maas-api"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857828    4860 cpu_manager.go:414] "containerMap: removing container" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857833    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857839    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857843    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857850    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857855    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857871    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857875    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857881    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857885    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857892    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b032a2d6-c0e0-46cf-9265-e214157142f2" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857897    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b032a2d6-c0e0-46cf-9265-e214157142f2" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857904    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857908    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857918    4860 cpu_manager.go:414] "containerMap: removing container" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857922    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857929    4860 cpu_manager.go:414] "containerMap: removing container" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857934    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857947    4860 cpu_manager.go:414] "containerMap: removing container" podUID="80f7cc19-3ccf-4e4a-8d30-8f76251885b0" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857952    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f7cc19-3ccf-4e4a-8d30-8f76251885b0" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857958    4860 cpu_manager.go:414] "containerMap: removing container" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" containerName="cleanup"
Oct 29 22:45:27.857942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857963    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857970    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857974    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857982    4860 cpu_manager.go:414] "containerMap: removing container" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857985    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857993    4860 cpu_manager.go:414] "containerMap: removing container" podUID="696b181a-c751-4741-8f95-86a613ef965c" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.857997    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696b181a-c751-4741-8f95-86a613ef965c" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858003    4860 cpu_manager.go:414] "containerMap: removing container" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858007    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858013    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8632936a-6257-4b14-9607-d7b18dfb85f7" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858017    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8632936a-6257-4b14-9607-d7b18dfb85f7" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858027    4860 cpu_manager.go:414] "containerMap: removing container" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858031    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858037    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e292294d-64b9-43b7-a0b9-89f63da16cdb" containerName="istio-proxy"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858043    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e292294d-64b9-43b7-a0b9-89f63da16cdb" containerName="istio-proxy"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858051    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8632936a-6257-4b14-9607-d7b18dfb85f7" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858055    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8632936a-6257-4b14-9607-d7b18dfb85f7" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858063    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858067    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858181    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="696b181a-c751-4741-8f95-86a613ef965c" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858188    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2580b08-b5bb-4306-a829-db8b7d832781" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858194    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858201    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8632936a-6257-4b14-9607-d7b18dfb85f7" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858207    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858213    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858220    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858226    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858231    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e292294d-64b9-43b7-a0b9-89f63da16cdb" containerName="istio-proxy"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858236    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858241    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858246    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="696b181a-c751-4741-8f95-86a613ef965c" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858250    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858257    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2580b08-b5bb-4306-a829-db8b7d832781" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858262    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f7cc19-3ccf-4e4a-8d30-8f76251885b0" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858269    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858274    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858280    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858286    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858292    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858297    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8632936a-6257-4b14-9607-d7b18dfb85f7" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858304    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b032a2d6-c0e0-46cf-9265-e214157142f2" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858309    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858314    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858318    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f2142b-4344-47d4-bdf0-40a45e61bb3e" containerName="maas-api"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858341    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858350    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858357    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858367    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858376    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858386    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b032a2d6-c0e0-46cf-9265-e214157142f2" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858396    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858402    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858411    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6184068-b2bf-487b-9b92-200dd0b4fd10" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858417    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="793ee6ac-d14d-480f-93d2-32bba5a4aed5" containerName="cleanup"
Oct 29 22:45:27.859482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.858422    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b032a2d6-c0e0-46cf-9265-e214157142f2" containerName="cleanup"
Oct 29 22:45:27.863959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.863940    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.882828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.882800    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"]
Oct 29 22:45:27.883610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.883577    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-certs\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.883731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.883622    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-envoy\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.883731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.883722    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istiod-ca-cert\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.883838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.883797    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-data\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.883963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.883943    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-token\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.884121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.884104    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbg4s\" (UniqueName: \"kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-kube-api-access-rbg4s\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.884190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.884152    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-socket\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.884397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.884311    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-credential-socket\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.884504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.884440    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-podinfo\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.890986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.890951    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8147a8-openshift-default-dockercfg-hmb4l\""
Oct 29 22:45:27.985526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.985487    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-credential-socket\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.985723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.985531    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-podinfo\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.985723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.985637    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-certs\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.985723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.985664    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-envoy\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.985723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.985702    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istiod-ca-cert\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.985976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.985779    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-data\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.985976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.985885    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-token\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.985976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.985904    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-credential-socket\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.986131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.985982    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-certs\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.986131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.986057    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbg4s\" (UniqueName: \"kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-kube-api-access-rbg4s\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.986131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.986105    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-socket\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.986290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.986208    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-data\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.986560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.986514    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-socket\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.986692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.986650    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istiod-ca-cert\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.988786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.988760    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-podinfo\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:27.988997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:27.988978    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-envoy\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:28.007353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.007310    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-token\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:28.007583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.007568    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbg4s\" (UniqueName: \"kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-kube-api-access-rbg4s\") pod \"e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") " pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:28.117495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.117417    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f97838e-68b7-4252-93dd-f535f4cca7f1" path="/var/lib/kubelet/pods/2f97838e-68b7-4252-93dd-f535f4cca7f1/volumes"
Oct 29 22:45:28.118022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.118006    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4" path="/var/lib/kubelet/pods/3b8a9038-c7c7-497b-b177-8c8342af29a4/volumes"
Oct 29 22:45:28.118467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.118454    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3" path="/var/lib/kubelet/pods/41a54691-ca33-41d5-bdf2-f3d603727ee3/volumes"
Oct 29 22:45:28.119251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.119234    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" path="/var/lib/kubelet/pods/6d1c0aa4-106c-44cf-8c66-c94717ba5c48/volumes"
Oct 29 22:45:28.119778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.119763    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e292294d-64b9-43b7-a0b9-89f63da16cdb" path="/var/lib/kubelet/pods/e292294d-64b9-43b7-a0b9-89f63da16cdb/volumes"
Oct 29 22:45:28.120183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.120169    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" path="/var/lib/kubelet/pods/f00c15ba-e735-40a2-a1be-f32f20d8d7be/volumes"
Oct 29 22:45:28.187668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.187631    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:28.515113 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:45:28.515067    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902c65e5_dea2_4afc_93c0_8b5b1b46bae6.slice/crio-09f8b3da6933abcdd2425e5b662c49613552e6d82c73c1615d583a1b66edbe8a WatchSource:0}: Error finding container 09f8b3da6933abcdd2425e5b662c49613552e6d82c73c1615d583a1b66edbe8a: Status 404 returned error can't find the container with id 09f8b3da6933abcdd2425e5b662c49613552e6d82c73c1615d583a1b66edbe8a
Oct 29 22:45:28.521369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.520911    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"]
Oct 29 22:45:28.521369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.520931    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:45:28.521369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.521092    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:45:28.521369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.521155    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:45:28.752948 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:28.752907    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:29.098416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:29.098382    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:29.419118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:29.418666    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" event={"ID":"902c65e5-dea2-4afc-93c0-8b5b1b46bae6","Type":"ContainerStarted","Data":"12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a"}
Oct 29 22:45:29.419961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:29.419773    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" event={"ID":"902c65e5-dea2-4afc-93c0-8b5b1b46bae6","Type":"ContainerStarted","Data":"09f8b3da6933abcdd2425e5b662c49613552e6d82c73c1615d583a1b66edbe8a"}
Oct 29 22:45:29.488464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:29.487545    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" podStartSLOduration=2.487522975 podStartE2EDuration="2.487522975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:29.483876744 +0000 UTC m=+1871.508652877" watchObservedRunningTime="2026-10-29 22:45:29.487522975 +0000 UTC m=+1871.512299107"
Oct 29 22:45:29.892352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:29.888805    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"]
Oct 29 22:45:30.037599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:30.037481    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:30.165148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:30.165079    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:30.190170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:30.190138    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:30.190542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:30.190518    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 29 22:45:30.190705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:30.190683    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" podUID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 29 22:45:30.565543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:30.565511    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:31.189714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:31.189677    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 29 22:45:31.190386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:31.190299    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" podUID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 29 22:45:31.440875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:31.440794    4860 generic.go:356] "Generic (PLEG): container finished" podID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerID="a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe" exitCode=0
Oct 29 22:45:31.440875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:31.440855    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" event={"ID":"307076b4-eceb-48f7-8ca6-5e11f666c439","Type":"ContainerDied","Data":"a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe"}
Oct 29 22:45:31.441276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:31.441262    4860 scope.go:122] "RemoveContainer" containerID="a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe"
Oct 29 22:45:31.443081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:31.443054    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:45:31.443155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:31.443126    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:45:31.443196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:31.443162    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:45:32.195375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:32.194719    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 29 22:45:32.195375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:32.194786    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" podUID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 29 22:45:32.213953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:32.213911    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:32.342299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:32.342262    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:32.505410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:32.502412    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" event={"ID":"307076b4-eceb-48f7-8ca6-5e11f666c439","Type":"ContainerStarted","Data":"355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea"}
Oct 29 22:45:33.193836 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:33.193789    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 29 22:45:33.194925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:33.194892    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" podUID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 29 22:45:33.353213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:33.353182    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:45:33.353640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:33.353349    4860 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 29 22:45:33.353640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:33.353383    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 29 22:45:33.940674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:33.940635    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:34.191191 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:34.190681    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 29 22:45:34.191191 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:34.190755    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" podUID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 29 22:45:34.352571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:34.352535    4860 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 29 22:45:34.352754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:34.352584    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 29 22:45:34.904584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:34.904537    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68" podUID="e40e4a62-bd4d-4df7-a185-efcaf2d1071f" containerName="payload-processing" containerID="cri-o://eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554" gracePeriod=25
Oct 29 22:45:35.191350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:35.188646    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 29 22:45:35.191350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:35.188716    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" podUID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 29 22:45:35.352922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:35.352886    4860 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 29 22:45:35.353094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:35.352940    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 29 22:45:35.622504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:35.622398    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:36.196943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:36.196902    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:36.197466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:36.197352    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:36.198031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:36.198014    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:45:36.352650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:36.352619    4860 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 29 22:45:36.352818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:36.352669    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 29 22:45:37.137506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:37.137474    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:37.352603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:37.352566    4860 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 29 22:45:37.352988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:37.352628    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 29 22:45:38.352532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:38.352489    4860 patch_prober.go:29] interesting pod/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 29 22:45:38.352752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:38.352563    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 29 22:45:38.696653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:38.696616    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:39.357774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.357744    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:45:39.358271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.358248    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:45:39.358917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.358897    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:45:39.376970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.376942    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"]
Oct 29 22:45:39.377902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.377883    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" containerName="cleanup"
Oct 29 22:45:39.377963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.377905    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00c15ba-e735-40a2-a1be-f32f20d8d7be" containerName="cleanup"
Oct 29 22:45:39.377963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.377919    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c2580b08-b5bb-4306-a829-db8b7d832781" containerName="cleanup"
Oct 29 22:45:39.377963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.377924    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2580b08-b5bb-4306-a829-db8b7d832781" containerName="cleanup"
Oct 29 22:45:39.377963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.377961    4860 cpu_manager.go:414] "containerMap: removing container" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3" containerName="cleanup"
Oct 29 22:45:39.378098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.377969    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3" containerName="cleanup"
Oct 29 22:45:39.378098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.377980    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" containerName="cleanup"
Oct 29 22:45:39.378098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.377985    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" containerName="cleanup"
Oct 29 22:45:39.378098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.377997    4860 cpu_manager.go:414] "containerMap: removing container" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" containerName="cleanup"
Oct 29 22:45:39.378098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378002    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" containerName="cleanup"
Oct 29 22:45:39.378098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378015    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b032a2d6-c0e0-46cf-9265-e214157142f2" containerName="cleanup"
Oct 29 22:45:39.378098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378020    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b032a2d6-c0e0-46cf-9265-e214157142f2" containerName="cleanup"
Oct 29 22:45:39.378098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378047    4860 cpu_manager.go:414] "containerMap: removing container" podUID="696b181a-c751-4741-8f95-86a613ef965c" containerName="cleanup"
Oct 29 22:45:39.378098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378055    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696b181a-c751-4741-8f95-86a613ef965c" containerName="cleanup"
Oct 29 22:45:39.378098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378074    4860 cpu_manager.go:414] "containerMap: removing container" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4" containerName="cleanup"
Oct 29 22:45:39.378098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378079    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4" containerName="cleanup"
Oct 29 22:45:39.378501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378203    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="696b181a-c751-4741-8f95-86a613ef965c" containerName="cleanup"
Oct 29 22:45:39.378501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378215    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2580b08-b5bb-4306-a829-db8b7d832781" containerName="cleanup"
Oct 29 22:45:39.378501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378226    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f7cc19-3ccf-4e4a-8d30-8f76251885b0" containerName="cleanup"
Oct 29 22:45:39.378501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378234    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8a9038-c7c7-497b-b177-8c8342af29a4" containerName="cleanup"
Oct 29 22:45:39.378501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378241    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="46509b71-e375-40ca-9a04-d35ddaa6e8bc" containerName="cleanup"
Oct 29 22:45:39.378501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378250    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a54691-ca33-41d5-bdf2-f3d603727ee3" containerName="cleanup"
Oct 29 22:45:39.378501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378258    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1c0aa4-106c-44cf-8c66-c94717ba5c48" containerName="cleanup"
Oct 29 22:45:39.378501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378264    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1276947c-a4b8-4eb3-8aaf-2d589e81b84c" containerName="cleanup"
Oct 29 22:45:39.378501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378274    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8632936a-6257-4b14-9607-d7b18dfb85f7" containerName="cleanup"
Oct 29 22:45:39.378501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.378283    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="455b3b28-47ae-40e5-a94d-c9ca25dc15c0" containerName="cleanup"
Oct 29 22:45:39.383088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.383050    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:39.399439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.399410    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:45:39.399568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.399453    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:45:39.400812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.400789    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"]
Oct 29 22:45:39.461412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.461365    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94b17d3f-x5c7p\" (UID: \"64573b13-0260-47f4-87b3-648f478f4f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:39.461412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.461415    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94b17d3f-x5c7p\" (UID: \"64573b13-0260-47f4-87b3-648f478f4f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:39.461840 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.461813    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zht5r\" (UniqueName: \"kubernetes.io/projected/64573b13-0260-47f4-87b3-648f478f4f34-kube-api-access-zht5r\") pod \"maas-api-revoke-sub-94b17d3f-x5c7p\" (UID: \"64573b13-0260-47f4-87b3-648f478f4f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:39.564028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.563997    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zht5r\" (UniqueName: \"kubernetes.io/projected/64573b13-0260-47f4-87b3-648f478f4f34-kube-api-access-zht5r\") pod \"maas-api-revoke-sub-94b17d3f-x5c7p\" (UID: \"64573b13-0260-47f4-87b3-648f478f4f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:39.564234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.564138    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94b17d3f-x5c7p\" (UID: \"64573b13-0260-47f4-87b3-648f478f4f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:39.564234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.564169    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94b17d3f-x5c7p\" (UID: \"64573b13-0260-47f4-87b3-648f478f4f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:39.564939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.564915    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94b17d3f-x5c7p\" (UID: \"64573b13-0260-47f4-87b3-648f478f4f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:39.566644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.566623    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94b17d3f-x5c7p\" (UID: \"64573b13-0260-47f4-87b3-648f478f4f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:39.587857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.587825    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zht5r\" (UniqueName: \"kubernetes.io/projected/64573b13-0260-47f4-87b3-648f478f4f34-kube-api-access-zht5r\") pod \"maas-api-revoke-sub-94b17d3f-x5c7p\" (UID: \"64573b13-0260-47f4-87b3-648f478f4f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:39.698337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:39.698299    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:40.027877 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:45:40.027846    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64573b13_0260_47f4_87b3_648f478f4f34.slice/crio-9b348bee3cf40ad4d69775583067992dec39de858d5c8aeefae854fb0db71973 WatchSource:0}: Error finding container 9b348bee3cf40ad4d69775583067992dec39de858d5c8aeefae854fb0db71973: Status 404 returned error can't find the container with id 9b348bee3cf40ad4d69775583067992dec39de858d5c8aeefae854fb0db71973
Oct 29 22:45:40.031890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:40.031867    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"]
Oct 29 22:45:40.563295 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:40.563258    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:40.670196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:40.670103    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p" event={"ID":"64573b13-0260-47f4-87b3-648f478f4f34","Type":"ContainerStarted","Data":"d2b8b743828eaa023976b1f29368b8179988b3c04d305a323e30770d60caa2af"}
Oct 29 22:45:40.670196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:40.670159    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p" event={"ID":"64573b13-0260-47f4-87b3-648f478f4f34","Type":"ContainerStarted","Data":"9b348bee3cf40ad4d69775583067992dec39de858d5c8aeefae854fb0db71973"}
Oct 29 22:45:40.706531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:40.706499    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:40.716298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:40.716251    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p" podStartSLOduration=1.7162374790000001 podStartE2EDuration="1.716237479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:40.71402014 +0000 UTC m=+1882.738796330" watchObservedRunningTime="2026-10-29 22:45:40.716237479 +0000 UTC m=+1882.741013624"
Oct 29 22:45:40.846080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:40.846051    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:41.679732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:41.679695    4860 generic.go:356] "Generic (PLEG): container finished" podID="64573b13-0260-47f4-87b3-648f478f4f34" containerID="d2b8b743828eaa023976b1f29368b8179988b3c04d305a323e30770d60caa2af" exitCode=0
Oct 29 22:45:41.680154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:41.679779    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p" event={"ID":"64573b13-0260-47f4-87b3-648f478f4f34","Type":"ContainerDied","Data":"d2b8b743828eaa023976b1f29368b8179988b3c04d305a323e30770d60caa2af"}
Oct 29 22:45:42.061150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:42.061114    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68" podUID="e40e4a62-bd4d-4df7-a185-efcaf2d1071f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:45:42.166646 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:42.166611    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:42.867385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:42.867361    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:42.920738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:42.920710    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-openshift-ca\") pod \"64573b13-0260-47f4-87b3-648f478f4f34\" (UID: \"64573b13-0260-47f4-87b3-648f478f4f34\") "
Oct 29 22:45:42.921029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:42.921005    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64573b13-0260-47f4-87b3-648f478f4f34-kube-api-access-zht5r\" (UniqueName: \"kubernetes.io/projected/64573b13-0260-47f4-87b3-648f478f4f34-kube-api-access-zht5r\") pod \"64573b13-0260-47f4-87b3-648f478f4f34\" (UID: \"64573b13-0260-47f4-87b3-648f478f4f34\") "
Oct 29 22:45:42.921128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:42.921061    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-openshift-ca" pod "64573b13-0260-47f4-87b3-648f478f4f34" (UID: "64573b13-0260-47f4-87b3-648f478f4f34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:42.921128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:42.921075    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-xks-ca\") pod \"64573b13-0260-47f4-87b3-648f478f4f34\" (UID: \"64573b13-0260-47f4-87b3-648f478f4f34\") "
Oct 29 22:45:42.921783 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:42.921759    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:42.923255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:42.923231    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64573b13-0260-47f4-87b3-648f478f4f34-kube-api-access-zht5r" pod "64573b13-0260-47f4-87b3-648f478f4f34" (UID: "64573b13-0260-47f4-87b3-648f478f4f34"). InnerVolumeSpecName "kube-api-access-zht5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:42.923650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:42.923624    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-xks-ca" pod "64573b13-0260-47f4-87b3-648f478f4f34" (UID: "64573b13-0260-47f4-87b3-648f478f4f34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:45:43.023100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:43.023039    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zht5r\" (UniqueName: \"kubernetes.io/projected/64573b13-0260-47f4-87b3-648f478f4f34-kube-api-access-zht5r\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:43.023100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:43.023065    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64573b13-0260-47f4-87b3-648f478f4f34-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:43.702066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:43.702030    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"
Oct 29 22:45:43.702253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:43.702070    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p" event={"ID":"64573b13-0260-47f4-87b3-648f478f4f34","Type":"ContainerDied","Data":"9b348bee3cf40ad4d69775583067992dec39de858d5c8aeefae854fb0db71973"}
Oct 29 22:45:43.702253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:43.702128    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b348bee3cf40ad4d69775583067992dec39de858d5c8aeefae854fb0db71973"
Oct 29 22:45:43.853553 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:43.853526    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:45.709494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:45.709459    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:47.457868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:47.457818    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:48.059224 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:48.059181    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:45:49.293046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:49.293015    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:51.138820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:51.138789    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:52.060106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:52.060062    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68" podUID="e40e4a62-bd4d-4df7-a185-efcaf2d1071f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:45:52.711292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:52.711248    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:54.241867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:54.241831    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:55.226995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.226963    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"
Oct 29 22:45:55.342774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.342739    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-kube-api-access-gg8sd\" (UniqueName: \"kubernetes.io/projected/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-kube-api-access-gg8sd\") pod \"e40e4a62-bd4d-4df7-a185-efcaf2d1071f\" (UID: \"e40e4a62-bd4d-4df7-a185-efcaf2d1071f\") "
Oct 29 22:45:55.343182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.342916    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-plugins-config-volume\") pod \"e40e4a62-bd4d-4df7-a185-efcaf2d1071f\" (UID: \"e40e4a62-bd4d-4df7-a185-efcaf2d1071f\") "
Oct 29 22:45:55.343240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.343214    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-plugins-config-volume" pod "e40e4a62-bd4d-4df7-a185-efcaf2d1071f" (UID: "e40e4a62-bd4d-4df7-a185-efcaf2d1071f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:55.343754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.343731    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:55.345273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.345241    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-kube-api-access-gg8sd" pod "e40e4a62-bd4d-4df7-a185-efcaf2d1071f" (UID: "e40e4a62-bd4d-4df7-a185-efcaf2d1071f"). InnerVolumeSpecName "kube-api-access-gg8sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:55.445933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.445871    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg8sd\" (UniqueName: \"kubernetes.io/projected/e40e4a62-bd4d-4df7-a185-efcaf2d1071f-kube-api-access-gg8sd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:55.572977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.572939    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:55.820292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.820203    4860 generic.go:356] "Generic (PLEG): container finished" podID="e40e4a62-bd4d-4df7-a185-efcaf2d1071f" containerID="eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554" exitCode=0
Oct 29 22:45:55.820292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.820251    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"
Oct 29 22:45:55.820582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.820287    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68" event={"ID":"e40e4a62-bd4d-4df7-a185-efcaf2d1071f","Type":"ContainerDied","Data":"eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554"}
Oct 29 22:45:55.820582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.820374    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68" event={"ID":"e40e4a62-bd4d-4df7-a185-efcaf2d1071f","Type":"ContainerDied","Data":"4169468c399e464ae9b6b05836470b3722b6045f77577d79169da34ee216ccbe"}
Oct 29 22:45:55.820582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.820406    4860 scope.go:122] "RemoveContainer" containerID="eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554"
Oct 29 22:45:55.834786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.834761    4860 scope.go:122] "RemoveContainer" containerID="eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554"
Oct 29 22:45:55.835100 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:45:55.835074    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554\": container with ID starting with eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554 not found: ID does not exist" containerID="eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554"
Oct 29 22:45:55.835155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.835115    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554"} err="failed to get container status \"eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554\": rpc error: code = NotFound desc = could not find container \"eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554\": container with ID starting with eb64a41b7801c65fe07b4b47fd384fb7419598fd9f4f25abfe467721deb82554 not found: ID does not exist"
Oct 29 22:45:55.880138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.880100    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"]
Oct 29 22:45:55.893145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:55.893108    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c296a3-544676fc5d-fvr68"]
Oct 29 22:45:56.060810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.060782    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:56.130864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.130762    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40e4a62-bd4d-4df7-a185-efcaf2d1071f" path="/var/lib/kubelet/pods/e40e4a62-bd4d-4df7-a185-efcaf2d1071f/volumes"
Oct 29 22:45:56.168600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.168559    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"]
Oct 29 22:45:56.169810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.169780    4860 cpu_manager.go:414] "containerMap: removing container" podUID="64573b13-0260-47f4-87b3-648f478f4f34" containerName="revoke-keys"
Oct 29 22:45:56.169810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.169802    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64573b13-0260-47f4-87b3-648f478f4f34" containerName="revoke-keys"
Oct 29 22:45:56.169960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.169854    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e40e4a62-bd4d-4df7-a185-efcaf2d1071f" containerName="payload-processing"
Oct 29 22:45:56.169960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.169860    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40e4a62-bd4d-4df7-a185-efcaf2d1071f" containerName="payload-processing"
Oct 29 22:45:56.170058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.170022    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40e4a62-bd4d-4df7-a185-efcaf2d1071f" containerName="payload-processing"
Oct 29 22:45:56.170058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.170039    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="64573b13-0260-47f4-87b3-648f478f4f34" containerName="revoke-keys"
Oct 29 22:45:56.176198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.176174    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:56.189411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.189389    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"]
Oct 29 22:45:56.192356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.192333    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:45:56.192510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.192493    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:45:56.261647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.261613    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d31392b-gtxmz\" (UID: \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:56.261815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.261793    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d31392b-gtxmz\" (UID: \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:56.261966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.261953    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk55n\" (UniqueName: \"kubernetes.io/projected/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-kube-api-access-dk55n\") pod \"maas-api-revoke-sub-8d31392b-gtxmz\" (UID: \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:56.362949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.362911    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d31392b-gtxmz\" (UID: \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:56.363309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.363051    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk55n\" (UniqueName: \"kubernetes.io/projected/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-kube-api-access-dk55n\") pod \"maas-api-revoke-sub-8d31392b-gtxmz\" (UID: \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:56.363309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.363188    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d31392b-gtxmz\" (UID: \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:56.363822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.363802    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d31392b-gtxmz\" (UID: \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:56.365443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.365420    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d31392b-gtxmz\" (UID: \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:56.384127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.384072    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk55n\" (UniqueName: \"kubernetes.io/projected/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-kube-api-access-dk55n\") pod \"maas-api-revoke-sub-8d31392b-gtxmz\" (UID: \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:56.493308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.493274    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:56.797995 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:45:56.797961    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b1b14a_c2db_44fe_9c95_8d0fd0ab90db.slice/crio-4e7eaf38134e686709a35b28de2849af0cbb2c84f34f7304ea7eca337bac65d8 WatchSource:0}: Error finding container 4e7eaf38134e686709a35b28de2849af0cbb2c84f34f7304ea7eca337bac65d8: Status 404 returned error can't find the container with id 4e7eaf38134e686709a35b28de2849af0cbb2c84f34f7304ea7eca337bac65d8
Oct 29 22:45:56.801239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.801217    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"]
Oct 29 22:45:56.831860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:56.831833    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz" event={"ID":"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db","Type":"ContainerStarted","Data":"4e7eaf38134e686709a35b28de2849af0cbb2c84f34f7304ea7eca337bac65d8"}
Oct 29 22:45:57.634119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:57.634073    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:57.843907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:57.843874    4860 generic.go:356] "Generic (PLEG): container finished" podID="71b1b14a-c2db-44fe-9c95-8d0fd0ab90db" containerID="a7bf51fab7b8cb122aaf4e9c5cb39f9db1091a73c240d60dc8cc9420ab06dcf1" exitCode=0
Oct 29 22:45:57.844039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:57.843961    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz" event={"ID":"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db","Type":"ContainerDied","Data":"a7bf51fab7b8cb122aaf4e9c5cb39f9db1091a73c240d60dc8cc9420ab06dcf1"}
Oct 29 22:45:59.028983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.028956    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:59.102671 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.102638    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-xks-ca\") pod \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\" (UID: \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\") "
Oct 29 22:45:59.102849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.102833    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-kube-api-access-dk55n\" (UniqueName: \"kubernetes.io/projected/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-kube-api-access-dk55n\") pod \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\" (UID: \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\") "
Oct 29 22:45:59.103169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.103081    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-openshift-ca\") pod \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\" (UID: \"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db\") "
Oct 29 22:45:59.104480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.103793    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-openshift-ca" pod "71b1b14a-c2db-44fe-9c95-8d0fd0ab90db" (UID: "71b1b14a-c2db-44fe-9c95-8d0fd0ab90db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:45:59.106597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.105602    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-xks-ca" pod "71b1b14a-c2db-44fe-9c95-8d0fd0ab90db" (UID: "71b1b14a-c2db-44fe-9c95-8d0fd0ab90db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:45:59.107817 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.106738    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-kube-api-access-dk55n" pod "71b1b14a-c2db-44fe-9c95-8d0fd0ab90db" (UID: "71b1b14a-c2db-44fe-9c95-8d0fd0ab90db"). InnerVolumeSpecName "kube-api-access-dk55n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:45:59.206430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.206403    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:59.206430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.206428    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk55n\" (UniqueName: \"kubernetes.io/projected/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-kube-api-access-dk55n\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:59.206619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.206437    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:45:59.278560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.278526    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:59.863025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.862998    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"
Oct 29 22:45:59.863221 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.863039    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz" event={"ID":"71b1b14a-c2db-44fe-9c95-8d0fd0ab90db","Type":"ContainerDied","Data":"4e7eaf38134e686709a35b28de2849af0cbb2c84f34f7304ea7eca337bac65d8"}
Oct 29 22:45:59.863221 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:45:59.863086    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e7eaf38134e686709a35b28de2849af0cbb2c84f34f7304ea7eca337bac65d8"
Oct 29 22:46:01.115282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:01.115251    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:02.718207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:02.718177    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:03.295333 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:03.295285    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:03.986271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:03.986234    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"]
Oct 29 22:46:03.986808 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:03.986490    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9" podUID="69fff808-0950-42f8-98dd-efb9c6a8fbac" containerName="maas-api" containerID="cri-o://d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264" gracePeriod=30
Oct 29 22:46:04.064464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.064435    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:04.346256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.345974    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:46:04.484241 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.484196    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-maas-api-tls\") pod \"69fff808-0950-42f8-98dd-efb9c6a8fbac\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") "
Oct 29 22:46:04.485061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.485034    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fff808-0950-42f8-98dd-efb9c6a8fbac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fff808-0950-42f8-98dd-efb9c6a8fbac-openshift-service-ca\") pod \"69fff808-0950-42f8-98dd-efb9c6a8fbac\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") "
Oct 29 22:46:04.485660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.485581    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fff808-0950-42f8-98dd-efb9c6a8fbac-openshift-service-ca" pod "69fff808-0950-42f8-98dd-efb9c6a8fbac" (UID: "69fff808-0950-42f8-98dd-efb9c6a8fbac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:46:04.485816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.485797    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-metrics-certs\") pod \"69fff808-0950-42f8-98dd-efb9c6a8fbac\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") "
Oct 29 22:46:04.485995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.485980    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fff808-0950-42f8-98dd-efb9c6a8fbac-kube-api-access-vvvwx\" (UniqueName: \"kubernetes.io/projected/69fff808-0950-42f8-98dd-efb9c6a8fbac-kube-api-access-vvvwx\") pod \"69fff808-0950-42f8-98dd-efb9c6a8fbac\" (UID: \"69fff808-0950-42f8-98dd-efb9c6a8fbac\") "
Oct 29 22:46:04.487742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.487720    4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fff808-0950-42f8-98dd-efb9c6a8fbac-openshift-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:04.502344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.488873    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fff808-0950-42f8-98dd-efb9c6a8fbac-kube-api-access-vvvwx" pod "69fff808-0950-42f8-98dd-efb9c6a8fbac" (UID: "69fff808-0950-42f8-98dd-efb9c6a8fbac"). InnerVolumeSpecName "kube-api-access-vvvwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:04.516516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.516063    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-maas-api-tls" pod "69fff808-0950-42f8-98dd-efb9c6a8fbac" (UID: "69fff808-0950-42f8-98dd-efb9c6a8fbac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:46:04.519371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.517418    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-metrics-certs" pod "69fff808-0950-42f8-98dd-efb9c6a8fbac" (UID: "69fff808-0950-42f8-98dd-efb9c6a8fbac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:46:04.590156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.590068    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-maas-api-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:04.590156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.590103    4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69fff808-0950-42f8-98dd-efb9c6a8fbac-metrics-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:04.590156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.590116    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvvwx\" (UniqueName: \"kubernetes.io/projected/69fff808-0950-42f8-98dd-efb9c6a8fbac-kube-api-access-vvvwx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:04.662363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.662285    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:04.796320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.796280    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:04.918781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.918694    4860 generic.go:356] "Generic (PLEG): container finished" podID="69fff808-0950-42f8-98dd-efb9c6a8fbac" containerID="d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264" exitCode=0
Oct 29 22:46:04.918781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.918745    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"
Oct 29 22:46:04.918781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.918753    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9" event={"ID":"69fff808-0950-42f8-98dd-efb9c6a8fbac","Type":"ContainerDied","Data":"d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264"}
Oct 29 22:46:04.919072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.918822    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9" event={"ID":"69fff808-0950-42f8-98dd-efb9c6a8fbac","Type":"ContainerDied","Data":"b3529908187f2616f172626076bc878f8cad14b10645742768795038d960fdaa"}
Oct 29 22:46:04.919072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.918848    4860 scope.go:122] "RemoveContainer" containerID="d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264"
Oct 29 22:46:04.930761 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.930693    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:04.934123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.934104    4860 scope.go:122] "RemoveContainer" containerID="d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264"
Oct 29 22:46:04.934429 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:46:04.934407    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264\": container with ID starting with d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264 not found: ID does not exist" containerID="d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264"
Oct 29 22:46:04.934499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.934440    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264"} err="failed to get container status \"d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264\": rpc error: code = NotFound desc = could not find container \"d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264\": container with ID starting with d0382126f6e95c5304338953f501ab58873803582ebbe4e00f64f3a4a457e264 not found: ID does not exist"
Oct 29 22:46:04.979655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.979624    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"]
Oct 29 22:46:04.992217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:04.992181    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c296a3-5788bcb888-lbgr9"]
Oct 29 22:46:05.299816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:05.299788    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:06.115934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:06.115616    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fff808-0950-42f8-98dd-efb9c6a8fbac" path="/var/lib/kubelet/pods/69fff808-0950-42f8-98dd-efb9c6a8fbac/volumes"
Oct 29 22:46:06.619993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:06.619953    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:07.445239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:07.445201    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:08.295916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:08.295885    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:09.823359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:09.823289    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:11.546110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:11.546076    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:12.282930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:12.282899    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:12.421779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:12.421743    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:12.599177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:12.599088    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"]
Oct 29 22:46:12.599680 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:12.599349    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9" podUID="46eab5e9-0a8b-4533-a604-d3478d41989b" containerName="istio-proxy" containerID="cri-o://a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9" gracePeriod=30
Oct 29 22:46:12.652813 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:12.652764    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:12.998839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:12.998794    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:13.135705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.135659    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:13.355598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.355490    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:13.476050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.476016    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:13.897698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.897665    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"]
Oct 29 22:46:13.898601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.898581    4860 cpu_manager.go:414] "containerMap: removing container" podUID="71b1b14a-c2db-44fe-9c95-8d0fd0ab90db" containerName="revoke-keys"
Oct 29 22:46:13.898601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.898601    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b1b14a-c2db-44fe-9c95-8d0fd0ab90db" containerName="revoke-keys"
Oct 29 22:46:13.898733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.898614    4860 cpu_manager.go:414] "containerMap: removing container" podUID="69fff808-0950-42f8-98dd-efb9c6a8fbac" containerName="maas-api"
Oct 29 22:46:13.898733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.898619    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fff808-0950-42f8-98dd-efb9c6a8fbac" containerName="maas-api"
Oct 29 22:46:13.898804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.898778    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b1b14a-c2db-44fe-9c95-8d0fd0ab90db" containerName="revoke-keys"
Oct 29 22:46:13.898804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.898787    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fff808-0950-42f8-98dd-efb9c6a8fbac" containerName="maas-api"
Oct 29 22:46:13.902786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.902769    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:13.919621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.919603    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-af260f-openshift-default-dockercfg-glq4f\""
Oct 29 22:46:13.942592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.942562    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"]
Oct 29 22:46:13.977953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:13.977923    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:14.090074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.090033    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-data\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.090281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.090223    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-credential-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.090281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.090258    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-certs\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.090467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.090309    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.090467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.090353    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istiod-ca-cert\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.090467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.090428    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-token\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.090642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.090464    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-podinfo\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.090642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.090509    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pck6b\" (UniqueName: \"kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-kube-api-access-pck6b\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.090642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.090612    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-envoy\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.156483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.155742    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.199401    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-credential-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.199441    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-certs\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.199477    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.199494    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istiod-ca-cert\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.199542    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-token\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.199564    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-podinfo\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.199593    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pck6b\" (UniqueName: \"kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-kube-api-access-pck6b\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.199656    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-envoy\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.199754    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-data\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.199970    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-credential-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.200078    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-data\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.200619    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-certs\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.200919    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.201986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.201385    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istiod-ca-cert\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.203007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.202978    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-envoy\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.205635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.205589    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-podinfo\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.224046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.224018    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-token\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.224387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.224303    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pck6b\" (UniqueName: \"kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-kube-api-access-pck6b\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.518709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.518674    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:14.858403 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:46:14.858364    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ce07ff_10ea_4294_bd37_aa7a12d9f4d0.slice/crio-630ecf9cb75353c6c6d4f49693b18d6d4c5f3480bc0e41787ab3392464f1d775 WatchSource:0}: Error finding container 630ecf9cb75353c6c6d4f49693b18d6d4c5f3480bc0e41787ab3392464f1d775: Status 404 returned error can't find the container with id 630ecf9cb75353c6c6d4f49693b18d6d4c5f3480bc0e41787ab3392464f1d775
Oct 29 22:46:14.861790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.861749    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:46:14.861900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.861822    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:46:14.861900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.861852    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:46:14.862249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:14.862218    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"]
Oct 29 22:46:15.030403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:15.030308    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:15.141046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:15.140953    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" event={"ID":"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0","Type":"ContainerStarted","Data":"e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70"}
Oct 29 22:46:15.141444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:15.141414    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" event={"ID":"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0","Type":"ContainerStarted","Data":"630ecf9cb75353c6c6d4f49693b18d6d4c5f3480bc0e41787ab3392464f1d775"}
Oct 29 22:46:15.221791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:15.221738    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" podStartSLOduration=2.221724844 podStartE2EDuration="2.221724844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:15.212744554 +0000 UTC m=+1917.237520702" watchObservedRunningTime="2026-10-29 22:46:15.221724844 +0000 UTC m=+1917.246500974"
Oct 29 22:46:15.519691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:15.519644    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:15.526420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:15.526393    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:16.152286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:16.152253    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:16.153398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:16.153372    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:46:17.044011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:17.043956    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:17.709756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:17.709713    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:17.743202 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:46:17.743168    4860 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46eab5e9_0a8b_4533_a604_d3478d41989b.slice/crio-conmon-a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:46:17.854206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:17.854166    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:17.928878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:17.928850    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:46:18.033216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.033184    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-socket\") pod \"46eab5e9-0a8b-4533-a604-d3478d41989b\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") "
Oct 29 22:46:18.033426 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.033318    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-credential-socket\") pod \"46eab5e9-0a8b-4533-a604-d3478d41989b\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") "
Oct 29 22:46:18.033502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.033490    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46eab5e9-0a8b-4533-a604-d3478d41989b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46eab5e9-0a8b-4533-a604-d3478d41989b-istiod-ca-cert\") pod \"46eab5e9-0a8b-4533-a604-d3478d41989b\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") "
Oct 29 22:46:18.033563 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.033522    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-kube-api-access-97c64\" (UniqueName: \"kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-kube-api-access-97c64\") pod \"46eab5e9-0a8b-4533-a604-d3478d41989b\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") "
Oct 29 22:46:18.033623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.033587    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-socket" pod "46eab5e9-0a8b-4533-a604-d3478d41989b" (UID: "46eab5e9-0a8b-4533-a604-d3478d41989b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.033743 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.033720    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-data\") pod \"46eab5e9-0a8b-4533-a604-d3478d41989b\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") "
Oct 29 22:46:18.033930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.033802    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-certs\") pod \"46eab5e9-0a8b-4533-a604-d3478d41989b\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") "
Oct 29 22:46:18.033930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.033856    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-envoy\") pod \"46eab5e9-0a8b-4533-a604-d3478d41989b\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") "
Oct 29 22:46:18.034076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.033942    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-token\" (UniqueName: \"kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-token\") pod \"46eab5e9-0a8b-4533-a604-d3478d41989b\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") "
Oct 29 22:46:18.034076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.033974    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-podinfo\") pod \"46eab5e9-0a8b-4533-a604-d3478d41989b\" (UID: \"46eab5e9-0a8b-4533-a604-d3478d41989b\") "
Oct 29 22:46:18.034076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.034004    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46eab5e9-0a8b-4533-a604-d3478d41989b-istiod-ca-cert" pod "46eab5e9-0a8b-4533-a604-d3478d41989b" (UID: "46eab5e9-0a8b-4533-a604-d3478d41989b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:46:18.034278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.034099    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-credential-socket" pod "46eab5e9-0a8b-4533-a604-d3478d41989b" (UID: "46eab5e9-0a8b-4533-a604-d3478d41989b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.034381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.034271    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-certs" pod "46eab5e9-0a8b-4533-a604-d3478d41989b" (UID: "46eab5e9-0a8b-4533-a604-d3478d41989b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.034381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.034359    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-data" pod "46eab5e9-0a8b-4533-a604-d3478d41989b" (UID: "46eab5e9-0a8b-4533-a604-d3478d41989b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.034906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.034880    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46eab5e9-0a8b-4533-a604-d3478d41989b-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.035025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.034910    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.035025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.034934    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.035025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.034955    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.035025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.034976    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.036663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.036633    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-kube-api-access-97c64" pod "46eab5e9-0a8b-4533-a604-d3478d41989b" (UID: "46eab5e9-0a8b-4533-a604-d3478d41989b"). InnerVolumeSpecName "kube-api-access-97c64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:18.037423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.037397    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-token" pod "46eab5e9-0a8b-4533-a604-d3478d41989b" (UID: "46eab5e9-0a8b-4533-a604-d3478d41989b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:18.037538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.037428    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-podinfo" pod "46eab5e9-0a8b-4533-a604-d3478d41989b" (UID: "46eab5e9-0a8b-4533-a604-d3478d41989b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:46:18.037951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.037923    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-envoy" pod "46eab5e9-0a8b-4533-a604-d3478d41989b" (UID: "46eab5e9-0a8b-4533-a604-d3478d41989b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:46:18.074810 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:46:18.074776    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:46:18.136236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.136203    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97c64\" (UniqueName: \"kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-kube-api-access-97c64\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.136375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.136240    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.136375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.136254    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.136375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.136267    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46eab5e9-0a8b-4533-a604-d3478d41989b-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:18.194748 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.194714    4860 generic.go:356] "Generic (PLEG): container finished" podID="46eab5e9-0a8b-4533-a604-d3478d41989b" containerID="a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9" exitCode=0
Oct 29 22:46:18.194950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.194763    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"
Oct 29 22:46:18.194950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.194793    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9" event={"ID":"46eab5e9-0a8b-4533-a604-d3478d41989b","Type":"ContainerDied","Data":"a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9"}
Oct 29 22:46:18.194950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.194852    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9" event={"ID":"46eab5e9-0a8b-4533-a604-d3478d41989b","Type":"ContainerDied","Data":"4cbb2ad1b0224dabdb442e350bac748878d04d356bc25c389e3021cc9c04d10c"}
Oct 29 22:46:18.194950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.194880    4860 scope.go:122] "RemoveContainer" containerID="a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9"
Oct 29 22:46:18.207857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.207833    4860 scope.go:122] "RemoveContainer" containerID="a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9"
Oct 29 22:46:18.208149 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:46:18.208124    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9\": container with ID starting with a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9 not found: ID does not exist" containerID="a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9"
Oct 29 22:46:18.208271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.208155    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9"} err="failed to get container status \"a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9\": rpc error: code = NotFound desc = could not find container \"a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9\": container with ID starting with a07dbc0dd7cf3c3e3122647997800f4fdc273aa1fe063ccc2dd1f6b6622fc8a9 not found: ID does not exist"
Oct 29 22:46:18.254933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.254895    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"]
Oct 29 22:46:18.260088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:18.260054    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c296a3-openshift-default-79f7f78789-6gwx9"]
Oct 29 22:46:19.013351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:19.010828    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:20.118516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:20.118480    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46eab5e9-0a8b-4533-a604-d3478d41989b" path="/var/lib/kubelet/pods/46eab5e9-0a8b-4533-a604-d3478d41989b/volumes"
Oct 29 22:46:20.171089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:20.171054    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:20.283304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:20.283270    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:20.848890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:20.848854    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:22.619978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:22.619925    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:22.668257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:22.668221    4860 scope.go:122] "RemoveContainer" containerID="f417ad30dc4d01dbd5a11ff17e855ecb7aacd767ca84a56f92b7081ed4b5f2cc"
Oct 29 22:46:22.681758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:22.681738    4860 scope.go:122] "RemoveContainer" containerID="2f180b9c7ae1a36984789c166bc995b5b894bd345e8ea0314210080fea7694d7"
Oct 29 22:46:22.696190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:22.695960    4860 scope.go:122] "RemoveContainer" containerID="3053d7411a3ca9ee2c536874119a14e875e2ddd6958037fb0853049743370768"
Oct 29 22:46:22.712543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:22.712517    4860 scope.go:122] "RemoveContainer" containerID="31991588a5583be1fd29de21600a5e5d63a633cdf98e35d6a8225128c072b4aa"
Oct 29 22:46:24.223510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:24.223477    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:25.824384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:25.823066    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:26.904059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.904019    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"]
Oct 29 22:46:26.905441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.905418    4860 cpu_manager.go:414] "containerMap: removing container" podUID="46eab5e9-0a8b-4533-a604-d3478d41989b" containerName="istio-proxy"
Oct 29 22:46:26.905525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.905444    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46eab5e9-0a8b-4533-a604-d3478d41989b" containerName="istio-proxy"
Oct 29 22:46:26.905687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.905672    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="46eab5e9-0a8b-4533-a604-d3478d41989b" containerName="istio-proxy"
Oct 29 22:46:26.912700 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.912674    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:26.928719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.928691    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"]
Oct 29 22:46:26.933941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.933915    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-6fc6bd\"/\"openshift-service-ca.crt\""
Oct 29 22:46:26.934077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.933940    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-6fc6bd\"/\"kube-root-ca.crt\""
Oct 29 22:46:26.942945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.942918    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-6fc6bd\"/\"default-dockercfg-nzgc4\""
Oct 29 22:46:26.943193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.943173    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-6fc6bd\"/\"auth-test-model-6fc6bd-kserve-self-signed-certs\""
Oct 29 22:46:26.981635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.981612    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:26.987143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.987116    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-dshm\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:26.987238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.987196    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tmp-dir\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:26.987296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.987285    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tls-certs\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:26.987481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.987448    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmz9n\" (UniqueName: \"kubernetes.io/projected/f6d6c31a-f447-4e47-9efa-4de8ea630a64-kube-api-access-gmz9n\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:26.987560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.987494    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-model-cache\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:26.987639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:26.987628    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-home\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.088593 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.088566    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-home\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.088716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.088626    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-dshm\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.088716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.088683    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tmp-dir\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.088809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.088733    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tls-certs\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.088853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.088840    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmz9n\" (UniqueName: \"kubernetes.io/projected/f6d6c31a-f447-4e47-9efa-4de8ea630a64-kube-api-access-gmz9n\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.088956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.088870    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-model-cache\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.089031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.088971    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-home\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.089221 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.089190    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tmp-dir\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.089372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.089266    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-model-cache\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.091038 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.091016    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-dshm\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.091306 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.091290    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tls-certs\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.109807 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.109782    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmz9n\" (UniqueName: \"kubernetes.io/projected/f6d6c31a-f447-4e47-9efa-4de8ea630a64-kube-api-access-gmz9n\") pod \"auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") " pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.232694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.232660    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:27.528802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.528728    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:27.586623 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:46:27.586569    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6d6c31a_f447_4e47_9efa_4de8ea630a64.slice/crio-db0b90633c42a54cd989816d79022c2ccd28c9abf48e50061760fe9be6803762 WatchSource:0}: Error finding container db0b90633c42a54cd989816d79022c2ccd28c9abf48e50061760fe9be6803762: Status 404 returned error can't find the container with id db0b90633c42a54cd989816d79022c2ccd28c9abf48e50061760fe9be6803762
Oct 29 22:46:27.592941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:27.592914    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"]
Oct 29 22:46:28.004290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.004248    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:28.173382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.170905    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"]
Oct 29 22:46:28.176992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.176721    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:28.193412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.193382    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:46:28.193924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.193899    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"]
Oct 29 22:46:28.210709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.209673    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:46:28.210709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.209829    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:46:28.210709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.210660    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:46:28.270057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.269956    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p888n\" (UniqueName: \"kubernetes.io/projected/87bdaa87-7553-4b70-a53d-331542828bee-kube-api-access-p888n\") pod \"maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf\" (UID: \"87bdaa87-7553-4b70-a53d-331542828bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:28.270057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.270030    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf\" (UID: \"87bdaa87-7553-4b70-a53d-331542828bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:28.270821 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.270773    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf\" (UID: \"87bdaa87-7553-4b70-a53d-331542828bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:28.315505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.315467    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb" event={"ID":"f6d6c31a-f447-4e47-9efa-4de8ea630a64","Type":"ContainerStarted","Data":"3cab11e6a7627b32ef3e8c9cd4143658c3a9500b9c0a29eb592b2eeae23beb7d"}
Oct 29 22:46:28.315505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.315510    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb" event={"ID":"f6d6c31a-f447-4e47-9efa-4de8ea630a64","Type":"ContainerStarted","Data":"db0b90633c42a54cd989816d79022c2ccd28c9abf48e50061760fe9be6803762"}
Oct 29 22:46:28.358707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.358664    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb" podStartSLOduration=2.358652431 podStartE2EDuration="2.358652431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:28.357128163 +0000 UTC m=+1930.381904294" watchObservedRunningTime="2026-10-29 22:46:28.358652431 +0000 UTC m=+1930.383428561"
Oct 29 22:46:28.372097 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.372071    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf\" (UID: \"87bdaa87-7553-4b70-a53d-331542828bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:28.372210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.372117    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p888n\" (UniqueName: \"kubernetes.io/projected/87bdaa87-7553-4b70-a53d-331542828bee-kube-api-access-p888n\") pod \"maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf\" (UID: \"87bdaa87-7553-4b70-a53d-331542828bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:28.372210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.372157    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf\" (UID: \"87bdaa87-7553-4b70-a53d-331542828bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:28.372822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.372797    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf\" (UID: \"87bdaa87-7553-4b70-a53d-331542828bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:28.374569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.374550    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf\" (UID: \"87bdaa87-7553-4b70-a53d-331542828bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:28.392716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.392695    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p888n\" (UniqueName: \"kubernetes.io/projected/87bdaa87-7553-4b70-a53d-331542828bee-kube-api-access-p888n\") pod \"maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf\" (UID: \"87bdaa87-7553-4b70-a53d-331542828bee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:28.522908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.522822    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:28.856810 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:46:28.856775    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87bdaa87_7553_4b70_a53d_331542828bee.slice/crio-8acf07fb1fbbbd6e608505524da9675d406b23508eb36bf6daeb8e3469781a95 WatchSource:0}: Error finding container 8acf07fb1fbbbd6e608505524da9675d406b23508eb36bf6daeb8e3469781a95: Status 404 returned error can't find the container with id 8acf07fb1fbbbd6e608505524da9675d406b23508eb36bf6daeb8e3469781a95
Oct 29 22:46:28.859981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:28.859956    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"]
Oct 29 22:46:29.233234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:29.233194    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:29.311077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:29.311043    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:29.329449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:29.329422    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf" event={"ID":"87bdaa87-7553-4b70-a53d-331542828bee","Type":"ContainerStarted","Data":"cc3fb3c2570b4a72187439268d5b40ee34ee2434ebd6d142646bb94f8e8b4642"}
Oct 29 22:46:29.329556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:29.329475    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf" event={"ID":"87bdaa87-7553-4b70-a53d-331542828bee","Type":"ContainerStarted","Data":"8acf07fb1fbbbd6e608505524da9675d406b23508eb36bf6daeb8e3469781a95"}
Oct 29 22:46:29.407612 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:29.407441    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf" podStartSLOduration=1.407419274 podStartE2EDuration="1.407419274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:29.404470775 +0000 UTC m=+1931.429246935" watchObservedRunningTime="2026-10-29 22:46:29.407419274 +0000 UTC m=+1931.432195406"
Oct 29 22:46:30.340552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:30.340517    4860 generic.go:356] "Generic (PLEG): container finished" podID="87bdaa87-7553-4b70-a53d-331542828bee" containerID="cc3fb3c2570b4a72187439268d5b40ee34ee2434ebd6d142646bb94f8e8b4642" exitCode=0
Oct 29 22:46:30.340983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:30.340594    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf" event={"ID":"87bdaa87-7553-4b70-a53d-331542828bee","Type":"ContainerDied","Data":"cc3fb3c2570b4a72187439268d5b40ee34ee2434ebd6d142646bb94f8e8b4642"}
Oct 29 22:46:31.160029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:31.159996    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:31.569381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:31.568795    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:31.725773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:31.725739    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-xks-ca\") pod \"87bdaa87-7553-4b70-a53d-331542828bee\" (UID: \"87bdaa87-7553-4b70-a53d-331542828bee\") "
Oct 29 22:46:31.725978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:31.725876    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87bdaa87-7553-4b70-a53d-331542828bee-kube-api-access-p888n\" (UniqueName: \"kubernetes.io/projected/87bdaa87-7553-4b70-a53d-331542828bee-kube-api-access-p888n\") pod \"87bdaa87-7553-4b70-a53d-331542828bee\" (UID: \"87bdaa87-7553-4b70-a53d-331542828bee\") "
Oct 29 22:46:31.725978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:31.725971    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-openshift-ca\") pod \"87bdaa87-7553-4b70-a53d-331542828bee\" (UID: \"87bdaa87-7553-4b70-a53d-331542828bee\") "
Oct 29 22:46:31.726403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:31.726368    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-openshift-ca" pod "87bdaa87-7553-4b70-a53d-331542828bee" (UID: "87bdaa87-7553-4b70-a53d-331542828bee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:46:31.726768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:31.726748    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:31.728456 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:31.728432    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-xks-ca" pod "87bdaa87-7553-4b70-a53d-331542828bee" (UID: "87bdaa87-7553-4b70-a53d-331542828bee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:46:31.728625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:31.728597    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87bdaa87-7553-4b70-a53d-331542828bee-kube-api-access-p888n" pod "87bdaa87-7553-4b70-a53d-331542828bee" (UID: "87bdaa87-7553-4b70-a53d-331542828bee"). InnerVolumeSpecName "kube-api-access-p888n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:46:31.828276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:31.828246    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p888n\" (UniqueName: \"kubernetes.io/projected/87bdaa87-7553-4b70-a53d-331542828bee-kube-api-access-p888n\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:31.828276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:31.828274    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bdaa87-7553-4b70-a53d-331542828bee-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:46:32.374531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:32.374501    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"
Oct 29 22:46:32.374768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:32.374535    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf" event={"ID":"87bdaa87-7553-4b70-a53d-331542828bee","Type":"ContainerDied","Data":"8acf07fb1fbbbd6e608505524da9675d406b23508eb36bf6daeb8e3469781a95"}
Oct 29 22:46:32.374768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:32.374623    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8acf07fb1fbbbd6e608505524da9675d406b23508eb36bf6daeb8e3469781a95"
Oct 29 22:46:32.862625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:32.862573    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:33.668291 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:33.668254    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:33.782855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:33.782793    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:34.617517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:34.617484    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:36.033124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:36.033094    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"]
Oct 29 22:46:36.047538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:36.047510    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-065d84b9-47cwd"]
Oct 29 22:46:36.114752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:36.114707    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb59bda-b92f-4c8c-8819-9c5a2ddc3ded" path="/var/lib/kubelet/pods/deb59bda-b92f-4c8c-8819-9c5a2ddc3ded/volumes"
Oct 29 22:46:36.123808 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:36.123784    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:37.233911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:37.233880    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:37.246734 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:37.246708    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:37.431229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:37.431196    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:46:37.697466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:37.697435    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:38.627102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:38.627068    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"]
Oct 29 22:46:38.663124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:38.663092    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"]
Oct 29 22:46:38.816221 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:38.816187    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:38.952842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:38.952805    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:39.466633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:39.464918    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:40.747273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.747236    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"]
Oct 29 22:46:40.748120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.748103    4860 cpu_manager.go:414] "containerMap: removing container" podUID="87bdaa87-7553-4b70-a53d-331542828bee" containerName="revoke-keys"
Oct 29 22:46:40.748167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.748121    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87bdaa87-7553-4b70-a53d-331542828bee" containerName="revoke-keys"
Oct 29 22:46:40.748260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.748251    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="87bdaa87-7553-4b70-a53d-331542828bee" containerName="revoke-keys"
Oct 29 22:46:40.753437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.753418    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.772037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.772013    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"]
Oct 29 22:46:40.780641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.780623    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5a69c9-openshift-default-dockercfg-w8dpp\""
Oct 29 22:46:40.879671 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.879638    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istiod-ca-cert\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.879846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.879692    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-socket\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.879846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.879775    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-data\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.879846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.879833    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-credential-socket\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.879993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.879902    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpglb\" (UniqueName: \"kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-kube-api-access-jpglb\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.879993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.879922    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-podinfo\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.879993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.879946    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-token\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.880120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.880050    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-certs\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.880164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.880134    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-envoy\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.981620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.981591    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-socket\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.981801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.981681    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-data\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.981801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.981712    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-credential-socket\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.981801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.981789    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpglb\" (UniqueName: \"kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-kube-api-access-jpglb\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.981986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.981829    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-podinfo\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.981986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.981862    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-token\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.981986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.981965    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-certs\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.982153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.982034    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-socket\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.982153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.982055    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-data\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.982153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.982096    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-envoy\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.982153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.982113    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-credential-socket\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.982449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.982185    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istiod-ca-cert\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.982449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.982291    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-certs\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.982800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.982781    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istiod-ca-cert\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.984209 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.984192    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-envoy\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:40.984524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:40.984508    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-podinfo\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:41.010875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:41.010818    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpglb\" (UniqueName: \"kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-kube-api-access-jpglb\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:41.011177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:41.011160    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-token\") pod \"e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") " pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:41.068442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:41.068414    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:41.140963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:41.140926    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:41.435002 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:46:41.434970    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087e8af1_5dcb_4f4a_8a0e_7ab3339eeb09.slice/crio-191069df5a66c19ff3d1e412480480de0a7b9cd2799e863c4e468cc71aaf716a WatchSource:0}: Error finding container 191069df5a66c19ff3d1e412480480de0a7b9cd2799e863c4e468cc71aaf716a: Status 404 returned error can't find the container with id 191069df5a66c19ff3d1e412480480de0a7b9cd2799e863c4e468cc71aaf716a
Oct 29 22:46:41.438620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:41.438585    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:46:41.438717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:41.438663    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:46:41.438755 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:41.438695    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:46:41.438944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:41.438916    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"]
Oct 29 22:46:41.482551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:41.482521    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d" event={"ID":"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09","Type":"ContainerStarted","Data":"191069df5a66c19ff3d1e412480480de0a7b9cd2799e863c4e468cc71aaf716a"}
Oct 29 22:46:42.504885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:42.504844    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d" event={"ID":"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09","Type":"ContainerStarted","Data":"f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf"}
Oct 29 22:46:42.592350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:42.591620    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d" podStartSLOduration=2.591599592 podStartE2EDuration="2.591599592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:42.563477592 +0000 UTC m=+1944.588253754" watchObservedRunningTime="2026-10-29 22:46:42.591599592 +0000 UTC m=+1944.616375728"
Oct 29 22:46:42.741286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:42.741255    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:43.087348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:43.069607    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:43.092346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:43.088410    4860 patch_prober.go:29] interesting pod/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 29 22:46:43.092346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:43.088467    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d" podUID="087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 29 22:46:43.628029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:43.627979    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl" podUID="54216655-1a8f-47c4-baed-e71d9b24460f" containerName="payload-processing" containerID="cri-o://82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888" gracePeriod=25
Oct 29 22:46:43.664177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:43.664148    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn" podUID="b11a66f5-937a-4e05-b359-0df3d2f43492" containerName="payload-pre-processing" containerID="cri-o://d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86" gracePeriod=25
Oct 29 22:46:44.092748 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:44.092705    4860 patch_prober.go:29] interesting pod/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 29 22:46:44.092927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:44.092773    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d" podUID="087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 29 22:46:44.464282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:44.464248    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:45.084350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:45.083648    4860 patch_prober.go:29] interesting pod/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 29 22:46:45.084350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:45.083711    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d" podUID="087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 29 22:46:45.992351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:45.991387    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:46.072351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:46.071729    4860 patch_prober.go:29] interesting pod/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 29 22:46:46.072351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:46.071811    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d" podUID="087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 29 22:46:47.077689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:47.077652    4860 patch_prober.go:29] interesting pod/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Oct 29 22:46:47.078284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:47.078250    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d" podUID="087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Oct 29 22:46:47.610219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:47.610182    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:48.066729 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:46:48.066694    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:46:48.074064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:48.074040    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:48.074560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:48.074542    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:48.075502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:48.075480    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:46:48.736951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:48.736910    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn" podUID="b11a66f5-937a-4e05-b359-0df3d2f43492" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:46:48.737456 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:48.736998    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl" podUID="54216655-1a8f-47c4-baed-e71d9b24460f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:46:49.342418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:49.342376    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:50.736206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:50.736176    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:51.261660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:51.261623    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:51.392582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:51.392542    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"]
Oct 29 22:46:51.397514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:51.397482    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:51.404862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:51.404840    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1750b-ndwcf"]
Oct 29 22:46:51.522637 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:51.522546    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:51.893424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:51.893349    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:52.114749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:52.114715    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87bdaa87-7553-4b70-a53d-331542828bee" path="/var/lib/kubelet/pods/87bdaa87-7553-4b70-a53d-331542828bee/volumes"
Oct 29 22:46:53.223416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:53.223383    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:54.832456 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:54.832424    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:56.495111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:56.495071    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:56.620349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:56.614862    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:56.730226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:56.730191    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:58.164919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:58.164857    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:58.358520 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:58.358484    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:58.492404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:58.492358    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:58.702621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:58.702591    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:58.735778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:58.735747    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl" podUID="54216655-1a8f-47c4-baed-e71d9b24460f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:46:58.735926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:58.735794    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn" podUID="b11a66f5-937a-4e05-b359-0df3d2f43492" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:46:59.183648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.183614    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"]
Oct 29 22:46:59.188422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.188401    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:46:59.213698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.213669    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:46:59.214011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.213993    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:46:59.224488 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.224454    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"]
Oct 29 22:46:59.297451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.297422    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:46:59.297599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.297505    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:46:59.337679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.337651    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4zs\" (UniqueName: \"kubernetes.io/projected/8cc44dba-a420-476b-978a-608c6d3b7c1b-kube-api-access-wc4zs\") pod \"maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6\" (UID: \"8cc44dba-a420-476b-978a-608c6d3b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:46:59.337792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.337780    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6\" (UID: \"8cc44dba-a420-476b-978a-608c6d3b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:46:59.337885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.337871    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6\" (UID: \"8cc44dba-a420-476b-978a-608c6d3b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:46:59.439610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.439532    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4zs\" (UniqueName: \"kubernetes.io/projected/8cc44dba-a420-476b-978a-608c6d3b7c1b-kube-api-access-wc4zs\") pod \"maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6\" (UID: \"8cc44dba-a420-476b-978a-608c6d3b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:46:59.439759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.439621    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6\" (UID: \"8cc44dba-a420-476b-978a-608c6d3b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:46:59.439759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.439689    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6\" (UID: \"8cc44dba-a420-476b-978a-608c6d3b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:46:59.440271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.440242    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6\" (UID: \"8cc44dba-a420-476b-978a-608c6d3b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:46:59.442054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.442033    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6\" (UID: \"8cc44dba-a420-476b-978a-608c6d3b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:46:59.460678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.460653    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4zs\" (UniqueName: \"kubernetes.io/projected/8cc44dba-a420-476b-978a-608c6d3b7c1b-kube-api-access-wc4zs\") pod \"maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6\" (UID: \"8cc44dba-a420-476b-978a-608c6d3b7c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:46:59.512890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.512868    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:46:59.820872 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:46:59.820841    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc44dba_a420_476b_978a_608c6d3b7c1b.slice/crio-69d84a54f4d61921a6b46812c4779ab62e41b431d7cdce0891a1dc0aaa018ff7 WatchSource:0}: Error finding container 69d84a54f4d61921a6b46812c4779ab62e41b431d7cdce0891a1dc0aaa018ff7: Status 404 returned error can't find the container with id 69d84a54f4d61921a6b46812c4779ab62e41b431d7cdce0891a1dc0aaa018ff7
Oct 29 22:46:59.824247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.824226    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"]
Oct 29 22:46:59.904231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:46:59.904209    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:00.709780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:00.709744    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6" event={"ID":"8cc44dba-a420-476b-978a-608c6d3b7c1b","Type":"ContainerStarted","Data":"119e83b5640517bf5773622836980e0aa4a7321f77eb046adaf0715679aa4b26"}
Oct 29 22:47:00.710224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:00.709795    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6" event={"ID":"8cc44dba-a420-476b-978a-608c6d3b7c1b","Type":"ContainerStarted","Data":"69d84a54f4d61921a6b46812c4779ab62e41b431d7cdce0891a1dc0aaa018ff7"}
Oct 29 22:47:00.754246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:00.754199    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6" podStartSLOduration=1.754186417 podStartE2EDuration="1.754186417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:00.750265934 +0000 UTC m=+1962.775042084" watchObservedRunningTime="2026-10-29 22:47:00.754186417 +0000 UTC m=+1962.778962577"
Oct 29 22:47:01.544894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:01.544810    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:01.720301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:01.720258    4860 generic.go:356] "Generic (PLEG): container finished" podID="8cc44dba-a420-476b-978a-608c6d3b7c1b" containerID="119e83b5640517bf5773622836980e0aa4a7321f77eb046adaf0715679aa4b26" exitCode=0
Oct 29 22:47:01.720722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:01.720305    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6" event={"ID":"8cc44dba-a420-476b-978a-608c6d3b7c1b","Type":"ContainerDied","Data":"119e83b5640517bf5773622836980e0aa4a7321f77eb046adaf0715679aa4b26"}
Oct 29 22:47:02.903814 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:02.903790    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:47:02.996624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:02.996543    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-openshift-ca\") pod \"8cc44dba-a420-476b-978a-608c6d3b7c1b\" (UID: \"8cc44dba-a420-476b-978a-608c6d3b7c1b\") "
Oct 29 22:47:02.996624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:02.996577    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-xks-ca\") pod \"8cc44dba-a420-476b-978a-608c6d3b7c1b\" (UID: \"8cc44dba-a420-476b-978a-608c6d3b7c1b\") "
Oct 29 22:47:02.996624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:02.996612    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc44dba-a420-476b-978a-608c6d3b7c1b-kube-api-access-wc4zs\" (UniqueName: \"kubernetes.io/projected/8cc44dba-a420-476b-978a-608c6d3b7c1b-kube-api-access-wc4zs\") pod \"8cc44dba-a420-476b-978a-608c6d3b7c1b\" (UID: \"8cc44dba-a420-476b-978a-608c6d3b7c1b\") "
Oct 29 22:47:02.996995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:02.996963    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-openshift-ca" pod "8cc44dba-a420-476b-978a-608c6d3b7c1b" (UID: "8cc44dba-a420-476b-978a-608c6d3b7c1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:02.997547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:02.997519    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:02.998986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:02.998952    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-xks-ca" pod "8cc44dba-a420-476b-978a-608c6d3b7c1b" (UID: "8cc44dba-a420-476b-978a-608c6d3b7c1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:47:02.999466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:02.999438    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc44dba-a420-476b-978a-608c6d3b7c1b-kube-api-access-wc4zs" pod "8cc44dba-a420-476b-978a-608c6d3b7c1b" (UID: "8cc44dba-a420-476b-978a-608c6d3b7c1b"). InnerVolumeSpecName "kube-api-access-wc4zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:03.099002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:03.098946    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc44dba-a420-476b-978a-608c6d3b7c1b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:03.099002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:03.099003    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc4zs\" (UniqueName: \"kubernetes.io/projected/8cc44dba-a420-476b-978a-608c6d3b7c1b-kube-api-access-wc4zs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:03.135217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:03.135185    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:03.739645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:03.739620    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"
Oct 29 22:47:03.739757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:03.739647    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6" event={"ID":"8cc44dba-a420-476b-978a-608c6d3b7c1b","Type":"ContainerDied","Data":"69d84a54f4d61921a6b46812c4779ab62e41b431d7cdce0891a1dc0aaa018ff7"}
Oct 29 22:47:03.739757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:03.739717    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69d84a54f4d61921a6b46812c4779ab62e41b431d7cdce0891a1dc0aaa018ff7"
Oct 29 22:47:03.973604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:03.973573    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"
Oct 29 22:47:04.017618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.017590    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"
Oct 29 22:47:04.116606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.116570    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54216655-1a8f-47c4-baed-e71d9b24460f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54216655-1a8f-47c4-baed-e71d9b24460f-plugins-config-volume\") pod \"54216655-1a8f-47c4-baed-e71d9b24460f\" (UID: \"54216655-1a8f-47c4-baed-e71d9b24460f\") "
Oct 29 22:47:04.116812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.116634    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54216655-1a8f-47c4-baed-e71d9b24460f-kube-api-access-6pdh4\" (UniqueName: \"kubernetes.io/projected/54216655-1a8f-47c4-baed-e71d9b24460f-kube-api-access-6pdh4\") pod \"54216655-1a8f-47c4-baed-e71d9b24460f\" (UID: \"54216655-1a8f-47c4-baed-e71d9b24460f\") "
Oct 29 22:47:04.116812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.116763    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11a66f5-937a-4e05-b359-0df3d2f43492-kube-api-access-tp5fd\" (UniqueName: \"kubernetes.io/projected/b11a66f5-937a-4e05-b359-0df3d2f43492-kube-api-access-tp5fd\") pod \"b11a66f5-937a-4e05-b359-0df3d2f43492\" (UID: \"b11a66f5-937a-4e05-b359-0df3d2f43492\") "
Oct 29 22:47:04.116939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.116857    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b11a66f5-937a-4e05-b359-0df3d2f43492-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11a66f5-937a-4e05-b359-0df3d2f43492-plugins-config-volume\") pod \"b11a66f5-937a-4e05-b359-0df3d2f43492\" (UID: \"b11a66f5-937a-4e05-b359-0df3d2f43492\") "
Oct 29 22:47:04.117301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.117275    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54216655-1a8f-47c4-baed-e71d9b24460f-plugins-config-volume" pod "54216655-1a8f-47c4-baed-e71d9b24460f" (UID: "54216655-1a8f-47c4-baed-e71d9b24460f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:04.117436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.117304    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11a66f5-937a-4e05-b359-0df3d2f43492-plugins-config-volume" pod "b11a66f5-937a-4e05-b359-0df3d2f43492" (UID: "b11a66f5-937a-4e05-b359-0df3d2f43492"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:04.117625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.117605    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54216655-1a8f-47c4-baed-e71d9b24460f-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.117699 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.117627    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11a66f5-937a-4e05-b359-0df3d2f43492-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.119045 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.119019    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54216655-1a8f-47c4-baed-e71d9b24460f-kube-api-access-6pdh4" pod "54216655-1a8f-47c4-baed-e71d9b24460f" (UID: "54216655-1a8f-47c4-baed-e71d9b24460f"). InnerVolumeSpecName "kube-api-access-6pdh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:04.120394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.120375    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11a66f5-937a-4e05-b359-0df3d2f43492-kube-api-access-tp5fd" pod "b11a66f5-937a-4e05-b359-0df3d2f43492" (UID: "b11a66f5-937a-4e05-b359-0df3d2f43492"). InnerVolumeSpecName "kube-api-access-tp5fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:04.219650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.219624    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp5fd\" (UniqueName: \"kubernetes.io/projected/b11a66f5-937a-4e05-b359-0df3d2f43492-kube-api-access-tp5fd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.219650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.219649    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pdh4\" (UniqueName: \"kubernetes.io/projected/54216655-1a8f-47c4-baed-e71d9b24460f-kube-api-access-6pdh4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:04.750226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.750190    4860 generic.go:356] "Generic (PLEG): container finished" podID="b11a66f5-937a-4e05-b359-0df3d2f43492" containerID="d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86" exitCode=0
Oct 29 22:47:04.750437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.750239    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"
Oct 29 22:47:04.750437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.750266    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn" event={"ID":"b11a66f5-937a-4e05-b359-0df3d2f43492","Type":"ContainerDied","Data":"d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86"}
Oct 29 22:47:04.750437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.750319    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn" event={"ID":"b11a66f5-937a-4e05-b359-0df3d2f43492","Type":"ContainerDied","Data":"126f395014df6b16525358fb81475af9a18a1949cda835354760ecca4c804d44"}
Oct 29 22:47:04.750437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.750356    4860 scope.go:122] "RemoveContainer" containerID="d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86"
Oct 29 22:47:04.752163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.752137    4860 generic.go:356] "Generic (PLEG): container finished" podID="54216655-1a8f-47c4-baed-e71d9b24460f" containerID="82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888" exitCode=0
Oct 29 22:47:04.752301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.752184    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"
Oct 29 22:47:04.752301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.752239    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl" event={"ID":"54216655-1a8f-47c4-baed-e71d9b24460f","Type":"ContainerDied","Data":"82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888"}
Oct 29 22:47:04.752435 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.752311    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl" event={"ID":"54216655-1a8f-47c4-baed-e71d9b24460f","Type":"ContainerDied","Data":"a0f4f2e6f14871410bdade232354649d67027b2c0b2d76ef095483513e31f610"}
Oct 29 22:47:04.767776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.767732    4860 scope.go:122] "RemoveContainer" containerID="d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86"
Oct 29 22:47:04.769357 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:47:04.768146    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86\": container with ID starting with d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86 not found: ID does not exist" containerID="d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86"
Oct 29 22:47:04.769357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.768181    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86"} err="failed to get container status \"d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86\": rpc error: code = NotFound desc = could not find container \"d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86\": container with ID starting with d8c1d19c1697da12616396e41cfdeadbc18c0fa046c40104ea4062bc28ce8e86 not found: ID does not exist"
Oct 29 22:47:04.769357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.768202    4860 scope.go:122] "RemoveContainer" containerID="82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888"
Oct 29 22:47:04.780155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.780138    4860 scope.go:122] "RemoveContainer" containerID="82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888"
Oct 29 22:47:04.780458 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:47:04.780437    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888\": container with ID starting with 82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888 not found: ID does not exist" containerID="82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888"
Oct 29 22:47:04.780530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.780464    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888"} err="failed to get container status \"82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888\": rpc error: code = NotFound desc = could not find container \"82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888\": container with ID starting with 82258c6015795b53c546e1b524dc0a5145e6a0150bf8d7a4426db66102628888 not found: ID does not exist"
Oct 29 22:47:04.815345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.813424    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"]
Oct 29 22:47:04.820842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.820819    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1750b-85f79d758d-n8sxl"]
Oct 29 22:47:04.842776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.842747    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:04.853055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.853029    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"]
Oct 29 22:47:04.864247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:04.864222    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1750b-65b7cbb459-gt6wn"]
Oct 29 22:47:06.197142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:06.197108    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54216655-1a8f-47c4-baed-e71d9b24460f" path="/var/lib/kubelet/pods/54216655-1a8f-47c4-baed-e71d9b24460f/volumes"
Oct 29 22:47:06.206749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:06.206718    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11a66f5-937a-4e05-b359-0df3d2f43492" path="/var/lib/kubelet/pods/b11a66f5-937a-4e05-b359-0df3d2f43492/volumes"
Oct 29 22:47:06.587664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:06.587562    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:08.101220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:08.101189    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:09.637177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:09.637144    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"]
Oct 29 22:47:09.672672 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:09.672645    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"]
Oct 29 22:47:09.814642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:09.814604    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:09.932753 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:09.932717    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:10.086059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:10.086020    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:11.524524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:11.524487    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:13.181080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:13.181030    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:13.291073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:13.291044    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:14.638405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:14.638364    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s" podUID="00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb" containerName="payload-processing" containerID="cri-o://7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81" gracePeriod=25
Oct 29 22:47:14.673934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:14.673904    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4" podUID="369bdcac-0746-4921-8a95-1a3006e4bc04" containerName="payload-pre-processing" containerID="cri-o://18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9" gracePeriod=25
Oct 29 22:47:14.942440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:14.942410    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:15.397679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:15.397599    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s" podUID="00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:47:15.397827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:15.397611    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4" podUID="369bdcac-0746-4921-8a95-1a3006e4bc04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:47:15.764897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:15.764861    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:15.891316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:15.891292    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:16.458297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:16.458266    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:18.061602 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:47:18.061567    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:47:18.061998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:18.061935    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:19.733520 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:19.733479    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:21.268500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:21.268465    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:22.901907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:22.901868    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:24.415593 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:24.415563    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:25.397067 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:25.397023    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4" podUID="369bdcac-0746-4921-8a95-1a3006e4bc04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:47:25.397249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:25.397040    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s" podUID="00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:47:26.025006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:26.024974    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:27.812586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:27.812549    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:29.309865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:29.309828    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:30.817052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:30.817011    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:32.427660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:32.427628    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:33.818078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.818042    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"]
Oct 29 22:47:33.818881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.818856    4860 cpu_manager.go:414] "containerMap: removing container" podUID="54216655-1a8f-47c4-baed-e71d9b24460f" containerName="payload-processing"
Oct 29 22:47:33.818881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.818881    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54216655-1a8f-47c4-baed-e71d9b24460f" containerName="payload-processing"
Oct 29 22:47:33.818994 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.818930    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b11a66f5-937a-4e05-b359-0df3d2f43492" containerName="payload-pre-processing"
Oct 29 22:47:33.818994 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.818937    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11a66f5-937a-4e05-b359-0df3d2f43492" containerName="payload-pre-processing"
Oct 29 22:47:33.818994 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.818954    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8cc44dba-a420-476b-978a-608c6d3b7c1b" containerName="revoke-keys"
Oct 29 22:47:33.818994 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.818959    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc44dba-a420-476b-978a-608c6d3b7c1b" containerName="revoke-keys"
Oct 29 22:47:33.819123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.819093    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11a66f5-937a-4e05-b359-0df3d2f43492" containerName="payload-pre-processing"
Oct 29 22:47:33.819123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.819102    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc44dba-a420-476b-978a-608c6d3b7c1b" containerName="revoke-keys"
Oct 29 22:47:33.819123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.819111    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="54216655-1a8f-47c4-baed-e71d9b24460f" containerName="payload-processing"
Oct 29 22:47:33.824589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.824568    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:33.839793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.839769    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"]
Oct 29 22:47:33.846749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.846660    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:47:33.846749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.846660    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:47:33.846749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.846705    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:47:33.854256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.854231    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:47:33.908393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.908365    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6df02d96-sx7nf\" (UID: \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:33.908519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.908491    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6df02d96-sx7nf\" (UID: \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:33.908653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:33.908630    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb78z\" (UniqueName: \"kubernetes.io/projected/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-kube-api-access-qb78z\") pod \"maas-api-revoke-sub-6df02d96-sx7nf\" (UID: \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:34.009680 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:34.009652    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb78z\" (UniqueName: \"kubernetes.io/projected/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-kube-api-access-qb78z\") pod \"maas-api-revoke-sub-6df02d96-sx7nf\" (UID: \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:34.009826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:34.009813    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6df02d96-sx7nf\" (UID: \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:34.009910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:34.009891    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6df02d96-sx7nf\" (UID: \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:34.010476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:34.010457    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6df02d96-sx7nf\" (UID: \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:34.012206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:34.012187    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6df02d96-sx7nf\" (UID: \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:34.029821 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:34.029792    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb78z\" (UniqueName: \"kubernetes.io/projected/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-kube-api-access-qb78z\") pod \"maas-api-revoke-sub-6df02d96-sx7nf\" (UID: \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:34.032413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:34.032391    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:34.140969 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:34.140873    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:34.468985 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:47:34.468953    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bb3e8a_2427_41a7_bdbd_f1b922bd3734.slice/crio-d90b6e8ad5c8c96d9a7590c5be30426876fc2ffe28220ee0e422f49d669b9a20 WatchSource:0}: Error finding container d90b6e8ad5c8c96d9a7590c5be30426876fc2ffe28220ee0e422f49d669b9a20: Status 404 returned error can't find the container with id d90b6e8ad5c8c96d9a7590c5be30426876fc2ffe28220ee0e422f49d669b9a20
Oct 29 22:47:34.473119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:34.473093    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"]
Oct 29 22:47:34.942548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:34.942521    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"
Oct 29 22:47:35.004593 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.004528    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"
Oct 29 22:47:35.026350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.025963    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369bdcac-0746-4921-8a95-1a3006e4bc04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369bdcac-0746-4921-8a95-1a3006e4bc04-plugins-config-volume\") pod \"369bdcac-0746-4921-8a95-1a3006e4bc04\" (UID: \"369bdcac-0746-4921-8a95-1a3006e4bc04\") "
Oct 29 22:47:35.026350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.026191    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369bdcac-0746-4921-8a95-1a3006e4bc04-kube-api-access-bmzhf\" (UniqueName: \"kubernetes.io/projected/369bdcac-0746-4921-8a95-1a3006e4bc04-kube-api-access-bmzhf\") pod \"369bdcac-0746-4921-8a95-1a3006e4bc04\" (UID: \"369bdcac-0746-4921-8a95-1a3006e4bc04\") "
Oct 29 22:47:35.026350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.026275    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369bdcac-0746-4921-8a95-1a3006e4bc04-plugins-config-volume" pod "369bdcac-0746-4921-8a95-1a3006e4bc04" (UID: "369bdcac-0746-4921-8a95-1a3006e4bc04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:35.027037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.027010    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369bdcac-0746-4921-8a95-1a3006e4bc04-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:35.028626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.028598    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369bdcac-0746-4921-8a95-1a3006e4bc04-kube-api-access-bmzhf" pod "369bdcac-0746-4921-8a95-1a3006e4bc04" (UID: "369bdcac-0746-4921-8a95-1a3006e4bc04"). InnerVolumeSpecName "kube-api-access-bmzhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:35.126094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.126063    4860 generic.go:356] "Generic (PLEG): container finished" podID="369bdcac-0746-4921-8a95-1a3006e4bc04" containerID="18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9" exitCode=0
Oct 29 22:47:35.126239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.126167    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"
Oct 29 22:47:35.126239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.126209    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4" event={"ID":"369bdcac-0746-4921-8a95-1a3006e4bc04","Type":"ContainerDied","Data":"18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9"}
Oct 29 22:47:35.126429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.126270    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4" event={"ID":"369bdcac-0746-4921-8a95-1a3006e4bc04","Type":"ContainerDied","Data":"1e443245a4b7b2ad703966976676e5acf6e49a615ba9cb43d293d346606f795f"}
Oct 29 22:47:35.126429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.126306    4860 scope.go:122] "RemoveContainer" containerID="18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9"
Oct 29 22:47:35.127586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.127563    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-kube-api-access-7gqq8\" (UniqueName: \"kubernetes.io/projected/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-kube-api-access-7gqq8\") pod \"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb\" (UID: \"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb\") "
Oct 29 22:47:35.127912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.127890    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-plugins-config-volume\") pod \"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb\" (UID: \"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb\") "
Oct 29 22:47:35.128035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.127984    4860 generic.go:356] "Generic (PLEG): container finished" podID="00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb" containerID="7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81" exitCode=0
Oct 29 22:47:35.128106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.128041    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"
Oct 29 22:47:35.128165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.128105    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s" event={"ID":"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb","Type":"ContainerDied","Data":"7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81"}
Oct 29 22:47:35.128227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.128166    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s" event={"ID":"00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb","Type":"ContainerDied","Data":"7121419f3fdfbe21f5241c3226c3bd2779a4624254680f492fbf1e562468cd58"}
Oct 29 22:47:35.128362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.128312    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-plugins-config-volume" pod "00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb" (UID: "00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:35.128839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.128810    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:35.128979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.128845    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmzhf\" (UniqueName: \"kubernetes.io/projected/369bdcac-0746-4921-8a95-1a3006e4bc04-kube-api-access-bmzhf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:35.130197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.130173    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-kube-api-access-7gqq8" pod "00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb" (UID: "00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb"). InnerVolumeSpecName "kube-api-access-7gqq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:35.130289 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.130215    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf" event={"ID":"25bb3e8a-2427-41a7-bdbd-f1b922bd3734","Type":"ContainerStarted","Data":"a7e0684d26a9dbc9a20c4400d57a443e7fe634edff56386dd2b80fd45b9c9adf"}
Oct 29 22:47:35.130289 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.130259    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf" event={"ID":"25bb3e8a-2427-41a7-bdbd-f1b922bd3734","Type":"ContainerStarted","Data":"d90b6e8ad5c8c96d9a7590c5be30426876fc2ffe28220ee0e422f49d669b9a20"}
Oct 29 22:47:35.152058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.151909    4860 scope.go:122] "RemoveContainer" containerID="18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9"
Oct 29 22:47:35.152191 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:47:35.152169    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9\": container with ID starting with 18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9 not found: ID does not exist" containerID="18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9"
Oct 29 22:47:35.152235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.152198    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9"} err="failed to get container status \"18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9\": rpc error: code = NotFound desc = could not find container \"18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9\": container with ID starting with 18641b85286bdb4f01d56b84f84d3d926574c05821dcc9a213e1173b77650bb9 not found: ID does not exist"
Oct 29 22:47:35.152235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.152215    4860 scope.go:122] "RemoveContainer" containerID="7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81"
Oct 29 22:47:35.162682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.162663    4860 scope.go:122] "RemoveContainer" containerID="7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81"
Oct 29 22:47:35.162936 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:47:35.162913    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81\": container with ID starting with 7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81 not found: ID does not exist" containerID="7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81"
Oct 29 22:47:35.162999 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.162937    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81"} err="failed to get container status \"7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81\": rpc error: code = NotFound desc = could not find container \"7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81\": container with ID starting with 7170a698c20672ea73fd00e77d12783c3baf2c944261e8ff732bf6e1ca293d81 not found: ID does not exist"
Oct 29 22:47:35.185726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.185686    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf" podStartSLOduration=2.185674647 podStartE2EDuration="2.185674647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:35.182451189 +0000 UTC m=+1997.207227321" watchObservedRunningTime="2026-10-29 22:47:35.185674647 +0000 UTC m=+1997.210450775"
Oct 29 22:47:35.230648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.230629    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gqq8\" (UniqueName: \"kubernetes.io/projected/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb-kube-api-access-7gqq8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:35.240873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.240847    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"]
Oct 29 22:47:35.246640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.246612    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8d985a-6b59cc474c-5vth4"]
Oct 29 22:47:35.480532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.480507    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"]
Oct 29 22:47:35.485582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.485558    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8d985a-78b6b6596-5tj4s"]
Oct 29 22:47:35.555307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.555286    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:35.656714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.656675    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"]
Oct 29 22:47:35.657611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.657592    4860 cpu_manager.go:414] "containerMap: removing container" podUID="00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb" containerName="payload-processing"
Oct 29 22:47:35.657611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.657611    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb" containerName="payload-processing"
Oct 29 22:47:35.657738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.657624    4860 cpu_manager.go:414] "containerMap: removing container" podUID="369bdcac-0746-4921-8a95-1a3006e4bc04" containerName="payload-pre-processing"
Oct 29 22:47:35.657738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.657629    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369bdcac-0746-4921-8a95-1a3006e4bc04" containerName="payload-pre-processing"
Oct 29 22:47:35.657803 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.657791    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb" containerName="payload-processing"
Oct 29 22:47:35.657803 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.657800    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="369bdcac-0746-4921-8a95-1a3006e4bc04" containerName="payload-pre-processing"
Oct 29 22:47:35.662619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.662599    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.681789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.679758    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"]
Oct 29 22:47:35.681789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.679990    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-6fc6bd-openshift-default-dockercfg-mff5d\""
Oct 29 22:47:35.755459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.755376    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwxvl\" (UniqueName: \"kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-kube-api-access-jwxvl\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.755601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.755499    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-data\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.755601 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.755594    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-credential-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.755682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.755646    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istiod-ca-cert\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.755774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.755752    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.755853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.755833    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-token\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.755889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.755879    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-podinfo\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.755981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.755965    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-envoy\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.756046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.756031    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-certs\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.857417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.857388    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-certs\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.857556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.857480    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwxvl\" (UniqueName: \"kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-kube-api-access-jwxvl\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.857556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.857539    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-data\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.857693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.857625    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-credential-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.857693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.857656    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istiod-ca-cert\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.857815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.857734    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.857815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.857788    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-certs\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.857934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.857802    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-token\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.857934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.857880    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-podinfo\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.857934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.857898    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-data\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.858099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.857944    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-credential-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.858099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.857952    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-envoy\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.858212 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.858127    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-socket\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.858557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.858529    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istiod-ca-cert\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.860180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.860158    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-envoy\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.860530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.860512    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-podinfo\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.878190 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.878160    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-token\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.878340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.878305    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwxvl\" (UniqueName: \"kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-kube-api-access-jwxvl\") pod \"e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") " pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:35.983088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:35.983061    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:36.118532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:36.118499    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb" path="/var/lib/kubelet/pods/00c1f27a-3fef-4e37-a7bd-9f4da9ea82fb/volumes"
Oct 29 22:47:36.119050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:36.119035    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369bdcac-0746-4921-8a95-1a3006e4bc04" path="/var/lib/kubelet/pods/369bdcac-0746-4921-8a95-1a3006e4bc04/volumes"
Oct 29 22:47:36.150721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:36.150687    4860 generic.go:356] "Generic (PLEG): container finished" podID="25bb3e8a-2427-41a7-bdbd-f1b922bd3734" containerID="a7e0684d26a9dbc9a20c4400d57a443e7fe634edff56386dd2b80fd45b9c9adf" exitCode=0
Oct 29 22:47:36.150896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:36.150770    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf" event={"ID":"25bb3e8a-2427-41a7-bdbd-f1b922bd3734","Type":"ContainerDied","Data":"a7e0684d26a9dbc9a20c4400d57a443e7fe634edff56386dd2b80fd45b9c9adf"}
Oct 29 22:47:36.299783 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:47:36.299754    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbb58b2_9058_48c8_8dc1_0e0a2d546ffa.slice/crio-4333ecd23ad76ba792ae5422011b0a750cac7bcee579bfdc77b763d4c32a0bc5 WatchSource:0}: Error finding container 4333ecd23ad76ba792ae5422011b0a750cac7bcee579bfdc77b763d4c32a0bc5: Status 404 returned error can't find the container with id 4333ecd23ad76ba792ae5422011b0a750cac7bcee579bfdc77b763d4c32a0bc5
Oct 29 22:47:36.302938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:36.302911    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"]
Oct 29 22:47:36.303478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:36.303440    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:47:36.303538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:36.303523    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:47:36.303618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:36.303563    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:47:37.231833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.231642    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj" event={"ID":"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa","Type":"ContainerStarted","Data":"50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19"}
Oct 29 22:47:37.231833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.231724    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj" event={"ID":"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa","Type":"ContainerStarted","Data":"4333ecd23ad76ba792ae5422011b0a750cac7bcee579bfdc77b763d4c32a0bc5"}
Oct 29 22:47:37.247370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.245892    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:37.311223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.306643    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj" podStartSLOduration=2.306623208 podStartE2EDuration="2.306623208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:37.291765221 +0000 UTC m=+1999.316541396" watchObservedRunningTime="2026-10-29 22:47:37.306623208 +0000 UTC m=+1999.331399339"
Oct 29 22:47:37.546097 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.546074    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:37.702768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.702720    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-xks-ca\") pod \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\" (UID: \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\") "
Oct 29 22:47:37.702978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.702776    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-kube-api-access-qb78z\" (UniqueName: \"kubernetes.io/projected/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-kube-api-access-qb78z\") pod \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\" (UID: \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\") "
Oct 29 22:47:37.703064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.702995    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-openshift-ca\") pod \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\" (UID: \"25bb3e8a-2427-41a7-bdbd-f1b922bd3734\") "
Oct 29 22:47:37.703455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.703420    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-openshift-ca" pod "25bb3e8a-2427-41a7-bdbd-f1b922bd3734" (UID: "25bb3e8a-2427-41a7-bdbd-f1b922bd3734"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:37.703904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.703879    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:37.706160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.706127    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-xks-ca" pod "25bb3e8a-2427-41a7-bdbd-f1b922bd3734" (UID: "25bb3e8a-2427-41a7-bdbd-f1b922bd3734"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:47:37.706562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.706538    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-kube-api-access-qb78z" pod "25bb3e8a-2427-41a7-bdbd-f1b922bd3734" (UID: "25bb3e8a-2427-41a7-bdbd-f1b922bd3734"). InnerVolumeSpecName "kube-api-access-qb78z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:37.805151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.805069    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:37.805151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.805099    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb78z\" (UniqueName: \"kubernetes.io/projected/25bb3e8a-2427-41a7-bdbd-f1b922bd3734-kube-api-access-qb78z\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:37.987752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.987678    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:37.992348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.989866    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 29 22:47:37.992348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:37.989926    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj" podUID="4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 29 22:47:38.240414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:38.240370    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf" event={"ID":"25bb3e8a-2427-41a7-bdbd-f1b922bd3734","Type":"ContainerDied","Data":"d90b6e8ad5c8c96d9a7590c5be30426876fc2ffe28220ee0e422f49d669b9a20"}
Oct 29 22:47:38.240414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:38.240411    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"
Oct 29 22:47:38.240878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:38.240454    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d90b6e8ad5c8c96d9a7590c5be30426876fc2ffe28220ee0e422f49d669b9a20"
Oct 29 22:47:38.889350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:38.879169    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:38.987926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:38.987437    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 29 22:47:38.987926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:38.987499    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj" podUID="4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 29 22:47:39.422262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:39.422221    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:39.854290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:39.854250    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:39.987349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:39.984611    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 29 22:47:39.987349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:39.984667    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj" podUID="4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 29 22:47:40.301934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:40.301893    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:40.446568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:40.446532    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:40.858617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:40.858546    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:41.000348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:40.994646    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 29 22:47:41.000348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:40.994712    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj" podUID="4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 29 22:47:41.988348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:41.986647    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 29 22:47:41.988348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:41.986711    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj" podUID="4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 29 22:47:42.533666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:42.533626    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:42.992459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:42.992417    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body=
Oct 29 22:47:42.992898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:42.992477    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj" podUID="4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused"
Oct 29 22:47:43.988909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:43.988878    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:43.989310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:43.989282    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:43.989937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:43.989918    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:47:43.992217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:43.992196    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:44.804173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:44.804136    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:44.922653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:44.922616    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"]
Oct 29 22:47:44.934834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:44.934807    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8d985a-cv7h6"]
Oct 29 22:47:44.976886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:44.976849    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:45.204606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:45.204573    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:45.379395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:45.379357    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:45.569439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:45.569367    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:46.114470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:46.114439    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc44dba-a420-476b-978a-608c6d3b7c1b" path="/var/lib/kubelet/pods/8cc44dba-a420-476b-978a-608c6d3b7c1b/volumes"
Oct 29 22:47:46.930275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:46.930246    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:47.101516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.101478    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:47.470342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.464868    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:47.793956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.793881    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"]
Oct 29 22:47:47.794788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.794770    4860 cpu_manager.go:414] "containerMap: removing container" podUID="25bb3e8a-2427-41a7-bdbd-f1b922bd3734" containerName="revoke-keys"
Oct 29 22:47:47.794847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.794790    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25bb3e8a-2427-41a7-bdbd-f1b922bd3734" containerName="revoke-keys"
Oct 29 22:47:47.794927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.794917    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="25bb3e8a-2427-41a7-bdbd-f1b922bd3734" containerName="revoke-keys"
Oct 29 22:47:47.799201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.799178    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:47.820092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.820070    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"]
Oct 29 22:47:47.826589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.826564    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:47:47.826697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.826631    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:47:47.861276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.861251    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:47:47.861510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.861488    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:47:47.968814 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.968785    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1d3035-z7f7z\" (UID: \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:47.968988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.968833    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1d3035-z7f7z\" (UID: \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:47.969131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:47.969107    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5md5\" (UniqueName: \"kubernetes.io/projected/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-kube-api-access-b5md5\") pod \"maas-api-revoke-sub-1b1d3035-z7f7z\" (UID: \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:48.059574 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:47:48.059490    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:47:48.070279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:48.070253    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5md5\" (UniqueName: \"kubernetes.io/projected/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-kube-api-access-b5md5\") pod \"maas-api-revoke-sub-1b1d3035-z7f7z\" (UID: \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:48.070434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:48.070411    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1d3035-z7f7z\" (UID: \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:48.070485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:48.070450    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1d3035-z7f7z\" (UID: \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:48.071089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:48.071064    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1d3035-z7f7z\" (UID: \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:48.072821 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:48.072799    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1d3035-z7f7z\" (UID: \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:48.092198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:48.092175    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5md5\" (UniqueName: \"kubernetes.io/projected/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-kube-api-access-b5md5\") pod \"maas-api-revoke-sub-1b1d3035-z7f7z\" (UID: \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:48.121708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:48.121685    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:48.433055 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:47:48.433024    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20b0cc5_7994_45bb_b2fe_f528be4ec7d6.slice/crio-6b753b5a6b6b7e22ce41ce0ad4f84c46b572067470aa4c2fe9246d5f3a405996 WatchSource:0}: Error finding container 6b753b5a6b6b7e22ce41ce0ad4f84c46b572067470aa4c2fe9246d5f3a405996: Status 404 returned error can't find the container with id 6b753b5a6b6b7e22ce41ce0ad4f84c46b572067470aa4c2fe9246d5f3a405996
Oct 29 22:47:48.439381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:48.439360    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"]
Oct 29 22:47:48.572137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:48.572106    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z" event={"ID":"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6","Type":"ContainerStarted","Data":"6b753b5a6b6b7e22ce41ce0ad4f84c46b572067470aa4c2fe9246d5f3a405996"}
Oct 29 22:47:48.737597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:48.737560    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:49.584126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:49.584098    4860 generic.go:356] "Generic (PLEG): container finished" podID="a20b0cc5-7994-45bb-b2fe-f528be4ec7d6" containerID="ae294e857339076e12e4e262088cd0af5612264dc593031f4a8e0aefd5dbbe0b" exitCode=0
Oct 29 22:47:49.584565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:49.584182    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z" event={"ID":"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6","Type":"ContainerDied","Data":"ae294e857339076e12e4e262088cd0af5612264dc593031f4a8e0aefd5dbbe0b"}
Oct 29 22:47:50.472039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:50.472002    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:50.792890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:50.792860    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:50.829529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:50.829501    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-xks-ca\") pod \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\" (UID: \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\") "
Oct 29 22:47:50.829655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:50.829567    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-kube-api-access-b5md5\" (UniqueName: \"kubernetes.io/projected/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-kube-api-access-b5md5\") pod \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\" (UID: \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\") "
Oct 29 22:47:50.829655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:50.829637    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-openshift-ca\") pod \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\" (UID: \"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6\") "
Oct 29 22:47:50.830017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:50.829991    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-openshift-ca" pod "a20b0cc5-7994-45bb-b2fe-f528be4ec7d6" (UID: "a20b0cc5-7994-45bb-b2fe-f528be4ec7d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:50.830491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:50.830472    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:50.831852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:50.831827    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-kube-api-access-b5md5" pod "a20b0cc5-7994-45bb-b2fe-f528be4ec7d6" (UID: "a20b0cc5-7994-45bb-b2fe-f528be4ec7d6"). InnerVolumeSpecName "kube-api-access-b5md5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:50.831852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:50.831833    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-xks-ca" pod "a20b0cc5-7994-45bb-b2fe-f528be4ec7d6" (UID: "a20b0cc5-7994-45bb-b2fe-f528be4ec7d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:47:50.932249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:50.932215    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:50.932249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:50.932247    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5md5\" (UniqueName: \"kubernetes.io/projected/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6-kube-api-access-b5md5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:51.605587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:51.605556    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"
Oct 29 22:47:51.605787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:51.605583    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z" event={"ID":"a20b0cc5-7994-45bb-b2fe-f528be4ec7d6","Type":"ContainerDied","Data":"6b753b5a6b6b7e22ce41ce0ad4f84c46b572067470aa4c2fe9246d5f3a405996"}
Oct 29 22:47:51.605787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:51.605634    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b753b5a6b6b7e22ce41ce0ad4f84c46b572067470aa4c2fe9246d5f3a405996"
Oct 29 22:47:52.091195 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:52.091128    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:52.219294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:52.219260    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:52.401222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:52.401135    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"]
Oct 29 22:47:52.401454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:52.401410    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy" containerID="cri-o://725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729" gracePeriod=30
Oct 29 22:47:52.452271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:52.452240    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:53.191071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:53.191034    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"]
Oct 29 22:47:53.191557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:53.191217    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq" podUID="d28324a2-e722-4488-81b2-dc8152adaf72" containerName="istio-proxy" containerID="cri-o://04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c" gracePeriod=30
Oct 29 22:47:53.281759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:53.281720    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:53.735356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:53.735282    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:53.877168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:53.873796    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:54.128926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:54.128850    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"]
Oct 29 22:47:54.129074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:54.129037    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy" containerID="cri-o://1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5" gracePeriod=30
Oct 29 22:47:54.312981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:54.312945    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"]
Oct 29 22:47:54.313394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:54.313134    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy" containerID="cri-o://0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65" gracePeriod=30
Oct 29 22:47:54.455970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:54.446485    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:55.631567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:55.631532    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:57.169736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.169707    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:57.714495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.714466    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:47:57.742094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.742068    4860 generic.go:356] "Generic (PLEG): container finished" podID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerID="725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729" exitCode=0
Oct 29 22:47:57.742263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.742135    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"
Oct 29 22:47:57.742263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.742160    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" event={"ID":"a3afafb7-388b-4114-82a8-adacbbb15f5d","Type":"ContainerDied","Data":"725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729"}
Oct 29 22:47:57.742263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.742249    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c" event={"ID":"a3afafb7-388b-4114-82a8-adacbbb15f5d","Type":"ContainerDied","Data":"a2db88d13d722ff69017f39927e45d64b9e1fd981dbc5adec86d84f86d975913"}
Oct 29 22:47:57.742491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.742280    4860 scope.go:122] "RemoveContainer" containerID="725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729"
Oct 29 22:47:57.756745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.756729    4860 scope.go:122] "RemoveContainer" containerID="de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b"
Oct 29 22:47:57.767525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.767499    4860 scope.go:122] "RemoveContainer" containerID="725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729"
Oct 29 22:47:57.767805 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:47:57.767783    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729\": container with ID starting with 725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729 not found: ID does not exist" containerID="725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729"
Oct 29 22:47:57.767880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.767813    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729"} err="failed to get container status \"725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729\": rpc error: code = NotFound desc = could not find container \"725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729\": container with ID starting with 725970849f05f5882ee78c01c075a39af01800a2f440612913042b296e005729 not found: ID does not exist"
Oct 29 22:47:57.767880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.767830    4860 scope.go:122] "RemoveContainer" containerID="de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b"
Oct 29 22:47:57.768084 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:47:57.768066    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b\": container with ID starting with de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b not found: ID does not exist" containerID="de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b"
Oct 29 22:47:57.768132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.768090    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b"} err="failed to get container status \"de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b\": rpc error: code = NotFound desc = could not find container \"de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b\": container with ID starting with de657963b14da4b4ca2f25e4b7ee1f3a32374330dab0871ac4da391253909b9b not found: ID does not exist"
Oct 29 22:47:57.827710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.827673    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-podinfo\") pod \"a3afafb7-388b-4114-82a8-adacbbb15f5d\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") "
Oct 29 22:47:57.827872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.827732    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3afafb7-388b-4114-82a8-adacbbb15f5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afafb7-388b-4114-82a8-adacbbb15f5d-istiod-ca-cert\") pod \"a3afafb7-388b-4114-82a8-adacbbb15f5d\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") "
Oct 29 22:47:57.827872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.827777    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-kube-api-access-rtn4k\" (UniqueName: \"kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-kube-api-access-rtn4k\") pod \"a3afafb7-388b-4114-82a8-adacbbb15f5d\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") "
Oct 29 22:47:57.827872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.827824    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-socket\") pod \"a3afafb7-388b-4114-82a8-adacbbb15f5d\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") "
Oct 29 22:47:57.828458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.828089    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-token\" (UniqueName: \"kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-token\") pod \"a3afafb7-388b-4114-82a8-adacbbb15f5d\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") "
Oct 29 22:47:57.828458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.828142    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-certs\") pod \"a3afafb7-388b-4114-82a8-adacbbb15f5d\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") "
Oct 29 22:47:57.828458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.828174    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-envoy\") pod \"a3afafb7-388b-4114-82a8-adacbbb15f5d\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") "
Oct 29 22:47:57.828458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.828228    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-credential-socket\") pod \"a3afafb7-388b-4114-82a8-adacbbb15f5d\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") "
Oct 29 22:47:57.828458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.828402    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-data\") pod \"a3afafb7-388b-4114-82a8-adacbbb15f5d\" (UID: \"a3afafb7-388b-4114-82a8-adacbbb15f5d\") "
Oct 29 22:47:57.828458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.828419    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-socket" pod "a3afafb7-388b-4114-82a8-adacbbb15f5d" (UID: "a3afafb7-388b-4114-82a8-adacbbb15f5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:57.828900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.828637    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-credential-socket" pod "a3afafb7-388b-4114-82a8-adacbbb15f5d" (UID: "a3afafb7-388b-4114-82a8-adacbbb15f5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:57.828900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.828828    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-data" pod "a3afafb7-388b-4114-82a8-adacbbb15f5d" (UID: "a3afafb7-388b-4114-82a8-adacbbb15f5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:57.829612 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.829586    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:57.829730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.829617    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:57.829730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.829632    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:57.830392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.830143    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-certs" pod "a3afafb7-388b-4114-82a8-adacbbb15f5d" (UID: "a3afafb7-388b-4114-82a8-adacbbb15f5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:57.830514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.830500    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3afafb7-388b-4114-82a8-adacbbb15f5d-istiod-ca-cert" pod "a3afafb7-388b-4114-82a8-adacbbb15f5d" (UID: "a3afafb7-388b-4114-82a8-adacbbb15f5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:57.832689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.832538    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-podinfo" pod "a3afafb7-388b-4114-82a8-adacbbb15f5d" (UID: "a3afafb7-388b-4114-82a8-adacbbb15f5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:47:57.832792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.832719    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-token" pod "a3afafb7-388b-4114-82a8-adacbbb15f5d" (UID: "a3afafb7-388b-4114-82a8-adacbbb15f5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:57.832792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.832765    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-kube-api-access-rtn4k" pod "a3afafb7-388b-4114-82a8-adacbbb15f5d" (UID: "a3afafb7-388b-4114-82a8-adacbbb15f5d"). InnerVolumeSpecName "kube-api-access-rtn4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:57.832920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.832787    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-envoy" pod "a3afafb7-388b-4114-82a8-adacbbb15f5d" (UID: "a3afafb7-388b-4114-82a8-adacbbb15f5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:57.931252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.931226    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:57.931252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.931250    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:57.931429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.931260    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:57.931429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.931268    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3afafb7-388b-4114-82a8-adacbbb15f5d-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:57.931429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.931278    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3afafb7-388b-4114-82a8-adacbbb15f5d-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:57.931429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:57.931286    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtn4k\" (UniqueName: \"kubernetes.io/projected/a3afafb7-388b-4114-82a8-adacbbb15f5d-kube-api-access-rtn4k\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:58.102742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.102707    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"]
Oct 29 22:47:58.119597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.119559    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8d985a-openshift-default-856f5f5bf5-w2k2c"]
Oct 29 22:47:58.511629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.511602    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:47:58.538016 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.537987    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-socket\") pod \"d28324a2-e722-4488-81b2-dc8152adaf72\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") "
Oct 29 22:47:58.538169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.538031    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d28324a2-e722-4488-81b2-dc8152adaf72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28324a2-e722-4488-81b2-dc8152adaf72-istio-podinfo\") pod \"d28324a2-e722-4488-81b2-dc8152adaf72\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") "
Oct 29 22:47:58.538169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.538085    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-envoy\") pod \"d28324a2-e722-4488-81b2-dc8152adaf72\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") "
Oct 29 22:47:58.538308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.538225    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-credential-socket\") pod \"d28324a2-e722-4488-81b2-dc8152adaf72\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") "
Oct 29 22:47:58.538308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.538241    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-socket" pod "d28324a2-e722-4488-81b2-dc8152adaf72" (UID: "d28324a2-e722-4488-81b2-dc8152adaf72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:58.538308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.538283    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28324a2-e722-4488-81b2-dc8152adaf72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28324a2-e722-4488-81b2-dc8152adaf72-istiod-ca-cert\") pod \"d28324a2-e722-4488-81b2-dc8152adaf72\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") "
Oct 29 22:47:58.538551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.538385    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-certs\") pod \"d28324a2-e722-4488-81b2-dc8152adaf72\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") "
Oct 29 22:47:58.538551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.538457    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-istio-token\" (UniqueName: \"kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-istio-token\") pod \"d28324a2-e722-4488-81b2-dc8152adaf72\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") "
Oct 29 22:47:58.538551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.538522    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-credential-socket" pod "d28324a2-e722-4488-81b2-dc8152adaf72" (UID: "d28324a2-e722-4488-81b2-dc8152adaf72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:58.538551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.538529    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-kube-api-access-pvqfs\" (UniqueName: \"kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-kube-api-access-pvqfs\") pod \"d28324a2-e722-4488-81b2-dc8152adaf72\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") "
Oct 29 22:47:58.538791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.538612    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-data\") pod \"d28324a2-e722-4488-81b2-dc8152adaf72\" (UID: \"d28324a2-e722-4488-81b2-dc8152adaf72\") "
Oct 29 22:47:58.538791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.538714    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-certs" pod "d28324a2-e722-4488-81b2-dc8152adaf72" (UID: "d28324a2-e722-4488-81b2-dc8152adaf72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:58.539478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.539300    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:58.539478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.539345    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:58.539478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.539359    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:58.539975 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.539943    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-data" pod "d28324a2-e722-4488-81b2-dc8152adaf72" (UID: "d28324a2-e722-4488-81b2-dc8152adaf72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:58.540146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.540123    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28324a2-e722-4488-81b2-dc8152adaf72-istiod-ca-cert" pod "d28324a2-e722-4488-81b2-dc8152adaf72" (UID: "d28324a2-e722-4488-81b2-dc8152adaf72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:58.540920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.540894    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-envoy" pod "d28324a2-e722-4488-81b2-dc8152adaf72" (UID: "d28324a2-e722-4488-81b2-dc8152adaf72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:58.541021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.540997    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-kube-api-access-pvqfs" pod "d28324a2-e722-4488-81b2-dc8152adaf72" (UID: "d28324a2-e722-4488-81b2-dc8152adaf72"). InnerVolumeSpecName "kube-api-access-pvqfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:58.541145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.541125    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d28324a2-e722-4488-81b2-dc8152adaf72-istio-podinfo" pod "d28324a2-e722-4488-81b2-dc8152adaf72" (UID: "d28324a2-e722-4488-81b2-dc8152adaf72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:47:58.541510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.541489    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-istio-token" pod "d28324a2-e722-4488-81b2-dc8152adaf72" (UID: "d28324a2-e722-4488-81b2-dc8152adaf72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:58.641230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.641164    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28324a2-e722-4488-81b2-dc8152adaf72-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:58.641230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.641190    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:58.641230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.641199    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28324a2-e722-4488-81b2-dc8152adaf72-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:58.641230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.641208    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:58.641230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.641218    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvqfs\" (UniqueName: \"kubernetes.io/projected/d28324a2-e722-4488-81b2-dc8152adaf72-kube-api-access-pvqfs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:58.641230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.641226    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28324a2-e722-4488-81b2-dc8152adaf72-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:58.752212 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.752177    4860 generic.go:356] "Generic (PLEG): container finished" podID="d28324a2-e722-4488-81b2-dc8152adaf72" containerID="04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c" exitCode=0
Oct 29 22:47:58.752388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.752230    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"
Oct 29 22:47:58.752388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.752259    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq" event={"ID":"d28324a2-e722-4488-81b2-dc8152adaf72","Type":"ContainerDied","Data":"04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c"}
Oct 29 22:47:58.752388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.752305    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq" event={"ID":"d28324a2-e722-4488-81b2-dc8152adaf72","Type":"ContainerDied","Data":"43bda2fee5f9dd10e6c1cd2acca6569c4075c1d5a4010f4945aa7934dcfaac1f"}
Oct 29 22:47:58.752388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.752360    4860 scope.go:122] "RemoveContainer" containerID="04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c"
Oct 29 22:47:58.763906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.763882    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:58.766030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.766014    4860 scope.go:122] "RemoveContainer" containerID="04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c"
Oct 29 22:47:58.766297 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:47:58.766277    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c\": container with ID starting with 04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c not found: ID does not exist" containerID="04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c"
Oct 29 22:47:58.766389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.766304    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c"} err="failed to get container status \"04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c\": rpc error: code = NotFound desc = could not find container \"04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c\": container with ID starting with 04e4ef2febfae8946bdc5f7e1fe868dd0225c50d46039928485a6eaa41b5bd4c not found: ID does not exist"
Oct 29 22:47:58.831868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.831846    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"]
Oct 29 22:47:58.842578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.842556    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-8cvvq"]
Oct 29 22:47:58.953339 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:58.953295    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:59.163389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.163362    4860 patch_prober.go:29] interesting pod/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:47:59.163493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.163411    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:47:59.168693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.168664    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:59.427027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.426977    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:47:59.453858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.453830    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istiod-ca-cert\") pod \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") "
Oct 29 22:47:59.453979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.453887    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-certs\") pod \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") "
Oct 29 22:47:59.453979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.453919    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-credential-socket\") pod \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") "
Oct 29 22:47:59.454263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.454050    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-socket\") pod \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") "
Oct 29 22:47:59.454263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.454077    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-envoy\") pod \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") "
Oct 29 22:47:59.454263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.454163    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-data\") pod \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") "
Oct 29 22:47:59.454263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.454230    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-token\" (UniqueName: \"kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-token\") pod \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") "
Oct 29 22:47:59.454533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.454282    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-podinfo\") pod \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") "
Oct 29 22:47:59.454533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.454266    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-certs" pod "722561f0-eacd-40b5-8d78-8a67b99bdf7f" (UID: "722561f0-eacd-40b5-8d78-8a67b99bdf7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:59.454533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.454320    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istiod-ca-cert" pod "722561f0-eacd-40b5-8d78-8a67b99bdf7f" (UID: "722561f0-eacd-40b5-8d78-8a67b99bdf7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:59.454533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.454386    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-socket" pod "722561f0-eacd-40b5-8d78-8a67b99bdf7f" (UID: "722561f0-eacd-40b5-8d78-8a67b99bdf7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:59.454533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.454406    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-credential-socket" pod "722561f0-eacd-40b5-8d78-8a67b99bdf7f" (UID: "722561f0-eacd-40b5-8d78-8a67b99bdf7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:59.454533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.454454    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-kube-api-access-7kn6r\" (UniqueName: \"kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-kube-api-access-7kn6r\") pod \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\" (UID: \"722561f0-eacd-40b5-8d78-8a67b99bdf7f\") "
Oct 29 22:47:59.454846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.454550    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-data" pod "722561f0-eacd-40b5-8d78-8a67b99bdf7f" (UID: "722561f0-eacd-40b5-8d78-8a67b99bdf7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:59.455185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.455163    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.455276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.455190    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.455276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.455203    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.455276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.455215    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.455276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.455228    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.456782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.456745    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-podinfo" pod "722561f0-eacd-40b5-8d78-8a67b99bdf7f" (UID: "722561f0-eacd-40b5-8d78-8a67b99bdf7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:47:59.456894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.456828    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-kube-api-access-7kn6r" pod "722561f0-eacd-40b5-8d78-8a67b99bdf7f" (UID: "722561f0-eacd-40b5-8d78-8a67b99bdf7f"). InnerVolumeSpecName "kube-api-access-7kn6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:59.457010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.456985    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-token" pod "722561f0-eacd-40b5-8d78-8a67b99bdf7f" (UID: "722561f0-eacd-40b5-8d78-8a67b99bdf7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:59.457100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.457063    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-envoy" pod "722561f0-eacd-40b5-8d78-8a67b99bdf7f" (UID: "722561f0-eacd-40b5-8d78-8a67b99bdf7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:59.557547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.557512    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.557547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.557548    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.557964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.557561    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kn6r\" (UniqueName: \"kubernetes.io/projected/722561f0-eacd-40b5-8d78-8a67b99bdf7f-kube-api-access-7kn6r\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.557964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.557575    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722561f0-eacd-40b5-8d78-8a67b99bdf7f-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.649538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.649481    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:47:59.760682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.760608    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-kube-api-access-g46rv\" (UniqueName: \"kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-kube-api-access-g46rv\") pod \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") "
Oct 29 22:47:59.760893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.760779    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-podinfo\") pod \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") "
Oct 29 22:47:59.760893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.760834    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-envoy\") pod \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") "
Oct 29 22:47:59.761029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.760895    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-certs\") pod \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") "
Oct 29 22:47:59.761029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.760936    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istiod-ca-cert\") pod \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") "
Oct 29 22:47:59.761029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.760973    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-data\") pod \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") "
Oct 29 22:47:59.761205 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.761029    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-socket\") pod \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") "
Oct 29 22:47:59.761205 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.761102    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-token\" (UniqueName: \"kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-token\") pod \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") "
Oct 29 22:47:59.761205 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.761177    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-credential-socket\") pod \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\" (UID: \"ee8f3210-ba1b-4981-bd2a-8ea5475773d5\") "
Oct 29 22:47:59.761397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.761250    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-certs" pod "ee8f3210-ba1b-4981-bd2a-8ea5475773d5" (UID: "ee8f3210-ba1b-4981-bd2a-8ea5475773d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:59.761565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.761541    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-data" pod "ee8f3210-ba1b-4981-bd2a-8ea5475773d5" (UID: "ee8f3210-ba1b-4981-bd2a-8ea5475773d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:59.761786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.761761    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-credential-socket" pod "ee8f3210-ba1b-4981-bd2a-8ea5475773d5" (UID: "ee8f3210-ba1b-4981-bd2a-8ea5475773d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:59.761991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.761970    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.762076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.761997    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.762076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.762012    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.762192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.762080    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istiod-ca-cert" pod "ee8f3210-ba1b-4981-bd2a-8ea5475773d5" (UID: "ee8f3210-ba1b-4981-bd2a-8ea5475773d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:47:59.762262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.762199    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-socket" pod "ee8f3210-ba1b-4981-bd2a-8ea5475773d5" (UID: "ee8f3210-ba1b-4981-bd2a-8ea5475773d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:59.764464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.764430    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-envoy" pod "ee8f3210-ba1b-4981-bd2a-8ea5475773d5" (UID: "ee8f3210-ba1b-4981-bd2a-8ea5475773d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:59.765015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.764991    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-podinfo" pod "ee8f3210-ba1b-4981-bd2a-8ea5475773d5" (UID: "ee8f3210-ba1b-4981-bd2a-8ea5475773d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:47:59.765110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.765077    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-kube-api-access-g46rv" pod "ee8f3210-ba1b-4981-bd2a-8ea5475773d5" (UID: "ee8f3210-ba1b-4981-bd2a-8ea5475773d5"). InnerVolumeSpecName "kube-api-access-g46rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:59.765172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.765153    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-token" pod "ee8f3210-ba1b-4981-bd2a-8ea5475773d5" (UID: "ee8f3210-ba1b-4981-bd2a-8ea5475773d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:59.768041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.768020    4860 generic.go:356] "Generic (PLEG): container finished" podID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerID="1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5" exitCode=0
Oct 29 22:47:59.768128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.768084    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"
Oct 29 22:47:59.768128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.768106    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" event={"ID":"722561f0-eacd-40b5-8d78-8a67b99bdf7f","Type":"ContainerDied","Data":"1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5"}
Oct 29 22:47:59.768262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.768175    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf" event={"ID":"722561f0-eacd-40b5-8d78-8a67b99bdf7f","Type":"ContainerDied","Data":"42e711e0d8874d2cbb94368329f0d2c5033fd32c6746b4490d195b8a52c05b05"}
Oct 29 22:47:59.768262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.768211    4860 scope.go:122] "RemoveContainer" containerID="1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5"
Oct 29 22:47:59.769827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.769801    4860 generic.go:356] "Generic (PLEG): container finished" podID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerID="0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65" exitCode=0
Oct 29 22:47:59.769912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.769866    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"
Oct 29 22:47:59.769912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.769888    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" event={"ID":"ee8f3210-ba1b-4981-bd2a-8ea5475773d5","Type":"ContainerDied","Data":"0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65"}
Oct 29 22:47:59.770030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.769951    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq" event={"ID":"ee8f3210-ba1b-4981-bd2a-8ea5475773d5","Type":"ContainerDied","Data":"29f5311a91b61c2ee2de8698f0780dd14b3bbbcfef5708be4adefcd1108f232a"}
Oct 29 22:47:59.783301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.783285    4860 scope.go:122] "RemoveContainer" containerID="1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5"
Oct 29 22:47:59.783612 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:47:59.783594    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5\": container with ID starting with 1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5 not found: ID does not exist" containerID="1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5"
Oct 29 22:47:59.783685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.783622    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5"} err="failed to get container status \"1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5\": rpc error: code = NotFound desc = could not find container \"1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5\": container with ID starting with 1619f46977d25367dac12f397bb0f9174a0b5b1614d5a3d77328af11ab3244c5 not found: ID does not exist"
Oct 29 22:47:59.783685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.783647    4860 scope.go:122] "RemoveContainer" containerID="0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65"
Oct 29 22:47:59.794056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.794040    4860 scope.go:122] "RemoveContainer" containerID="0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65"
Oct 29 22:47:59.794333 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:47:59.794301    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65\": container with ID starting with 0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65 not found: ID does not exist" containerID="0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65"
Oct 29 22:47:59.794407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.794365    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65"} err="failed to get container status \"0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65\": rpc error: code = NotFound desc = could not find container \"0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65\": container with ID starting with 0855c57e6649c78240f18495b04bb7020afe233ff4f980570ae05785e57b3e65 not found: ID does not exist"
Oct 29 22:47:59.828493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.828462    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"]
Oct 29 22:47:59.833468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.833443    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-nf9qq"]
Oct 29 22:47:59.863281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.863254    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.863281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.863282    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.863281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.863291    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.863476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.863301    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.863476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.863310    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.863476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.863319    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g46rv\" (UniqueName: \"kubernetes.io/projected/ee8f3210-ba1b-4981-bd2a-8ea5475773d5-kube-api-access-g46rv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:47:59.869576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.869551    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"]
Oct 29 22:47:59.873755 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:47:59.873731    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-lqhnf"]
Oct 29 22:48:00.114622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.114533    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" path="/var/lib/kubelet/pods/722561f0-eacd-40b5-8d78-8a67b99bdf7f/volumes"
Oct 29 22:48:00.115185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.115161    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" path="/var/lib/kubelet/pods/a3afafb7-388b-4114-82a8-adacbbb15f5d/volumes"
Oct 29 22:48:00.115767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.115744    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28324a2-e722-4488-81b2-dc8152adaf72" path="/var/lib/kubelet/pods/d28324a2-e722-4488-81b2-dc8152adaf72/volumes"
Oct 29 22:48:00.116641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.116620    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" path="/var/lib/kubelet/pods/ee8f3210-ba1b-4981-bd2a-8ea5475773d5/volumes"
Oct 29 22:48:00.431057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.431010    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:00.519008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.518973    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"]
Oct 29 22:48:00.519834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.519818    4860 cpu_manager.go:414] "containerMap: removing container" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy"
Oct 29 22:48:00.519890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.519835    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy"
Oct 29 22:48:00.519890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.519847    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a20b0cc5-7994-45bb-b2fe-f528be4ec7d6" containerName="revoke-keys"
Oct 29 22:48:00.519890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.519853    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a20b0cc5-7994-45bb-b2fe-f528be4ec7d6" containerName="revoke-keys"
Oct 29 22:48:00.519890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.519867    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy"
Oct 29 22:48:00.519890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.519872    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy"
Oct 29 22:48:00.519890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.519879    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d28324a2-e722-4488-81b2-dc8152adaf72" containerName="istio-proxy"
Oct 29 22:48:00.519890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.519884    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28324a2-e722-4488-81b2-dc8152adaf72" containerName="istio-proxy"
Oct 29 22:48:00.519890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.519898    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy"
Oct 29 22:48:00.519890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.519902    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy"
Oct 29 22:48:00.520182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.519911    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy"
Oct 29 22:48:00.520182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.519916    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy"
Oct 29 22:48:00.520182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.520024    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="722561f0-eacd-40b5-8d78-8a67b99bdf7f" containerName="istio-proxy"
Oct 29 22:48:00.520182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.520033    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a20b0cc5-7994-45bb-b2fe-f528be4ec7d6" containerName="revoke-keys"
Oct 29 22:48:00.520182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.520041    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee8f3210-ba1b-4981-bd2a-8ea5475773d5" containerName="istio-proxy"
Oct 29 22:48:00.520182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.520047    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28324a2-e722-4488-81b2-dc8152adaf72" containerName="istio-proxy"
Oct 29 22:48:00.520182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.520053    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy"
Oct 29 22:48:00.520182 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.520060    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3afafb7-388b-4114-82a8-adacbbb15f5d" containerName="istio-proxy"
Oct 29 22:48:00.525712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.525692    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"
Oct 29 22:48:00.543051 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.543030    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f8ca96\""
Oct 29 22:48:00.545278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.545255    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"]
Oct 29 22:48:00.566708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.566680    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"]
Oct 29 22:48:00.572272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.572249    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"
Oct 29 22:48:00.575389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.575358    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwv96\" (UniqueName: \"kubernetes.io/projected/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-kube-api-access-gwv96\") pod \"payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m\" (UID: \"d55a1f4c-beeb-4c8d-bccb-d07d86dda333\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"
Oct 29 22:48:00.575547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.575529    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m\" (UID: \"d55a1f4c-beeb-4c8d-bccb-d07d86dda333\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"
Oct 29 22:48:00.592786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.592753    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f8ca96-dockercfg-9lt9h\""
Oct 29 22:48:00.593893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.593830    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:00.598134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.598043    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"]
Oct 29 22:48:00.677298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.677264    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwv96\" (UniqueName: \"kubernetes.io/projected/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-kube-api-access-gwv96\") pod \"payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m\" (UID: \"d55a1f4c-beeb-4c8d-bccb-d07d86dda333\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"
Oct 29 22:48:00.677528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.677370    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72\" (UID: \"2c8afc25-6b5d-4136-8b69-0ef94a9ae866\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"
Oct 29 22:48:00.677528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.677485    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m\" (UID: \"d55a1f4c-beeb-4c8d-bccb-d07d86dda333\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"
Oct 29 22:48:00.677625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.677571    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdr94\" (UniqueName: \"kubernetes.io/projected/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-kube-api-access-tdr94\") pod \"payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72\" (UID: \"2c8afc25-6b5d-4136-8b69-0ef94a9ae866\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"
Oct 29 22:48:00.678134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.678112    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m\" (UID: \"d55a1f4c-beeb-4c8d-bccb-d07d86dda333\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"
Oct 29 22:48:00.698282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.698213    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwv96\" (UniqueName: \"kubernetes.io/projected/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-kube-api-access-gwv96\") pod \"payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m\" (UID: \"d55a1f4c-beeb-4c8d-bccb-d07d86dda333\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"
Oct 29 22:48:00.836767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.835949    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdr94\" (UniqueName: \"kubernetes.io/projected/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-kube-api-access-tdr94\") pod \"payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72\" (UID: \"2c8afc25-6b5d-4136-8b69-0ef94a9ae866\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"
Oct 29 22:48:00.836767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.836199    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72\" (UID: \"2c8afc25-6b5d-4136-8b69-0ef94a9ae866\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"
Oct 29 22:48:00.840348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.838101    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72\" (UID: \"2c8afc25-6b5d-4136-8b69-0ef94a9ae866\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"
Oct 29 22:48:00.844349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.842618    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"
Oct 29 22:48:00.896529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.896494    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdr94\" (UniqueName: \"kubernetes.io/projected/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-kube-api-access-tdr94\") pod \"payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72\" (UID: \"2c8afc25-6b5d-4136-8b69-0ef94a9ae866\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"
Oct 29 22:48:00.933877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:00.933803    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:01.092664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.092574    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"]
Oct 29 22:48:01.126905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.125825    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"]
Oct 29 22:48:01.127312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.127284    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:01.153586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.148899    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:48:01.153586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.149239    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:48:01.153586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.149300    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:48:01.153586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.149385    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:48:01.193402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.192422    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc\" (UID: \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:01.193402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.192566    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stn85\" (UniqueName: \"kubernetes.io/projected/40a7b7f8-7f10-445f-8e90-af03ec8f8498-kube-api-access-stn85\") pod \"maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc\" (UID: \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:01.193402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.192620    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc\" (UID: \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:01.194853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.194826    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"
Oct 29 22:48:01.295013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.294974    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stn85\" (UniqueName: \"kubernetes.io/projected/40a7b7f8-7f10-445f-8e90-af03ec8f8498-kube-api-access-stn85\") pod \"maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc\" (UID: \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:01.295135 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.295049    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc\" (UID: \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:01.295319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.295296    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc\" (UID: \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:01.296025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.296002    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc\" (UID: \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:01.310365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.310036    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc\" (UID: \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:01.319528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.319500    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stn85\" (UniqueName: \"kubernetes.io/projected/40a7b7f8-7f10-445f-8e90-af03ec8f8498-kube-api-access-stn85\") pod \"maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc\" (UID: \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:01.336696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.336664    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:01.368533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.368510    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"]
Oct 29 22:48:01.387861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.387810    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"]
Oct 29 22:48:01.390477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.390372    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" podUID="0357f513-e857-424f-91f8-130b056ca135" containerName="istio-proxy" containerID="cri-o://e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30" gracePeriod=30
Oct 29 22:48:01.398552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.398528    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"]
Oct 29 22:48:01.398776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.398745    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerName="istio-proxy" containerID="cri-o://dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee" gracePeriod=30
Oct 29 22:48:01.462186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.462160    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:01.618444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.617773    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"]
Oct 29 22:48:01.849935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.849890    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m" event={"ID":"d55a1f4c-beeb-4c8d-bccb-d07d86dda333","Type":"ContainerStarted","Data":"99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4"}
Oct 29 22:48:01.850140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.850000    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m" event={"ID":"d55a1f4c-beeb-4c8d-bccb-d07d86dda333","Type":"ContainerStarted","Data":"1444578c6d680b549398aee2b1fdd5c64e41cdda548c9a71bb6453e6d016aca2"}
Oct 29 22:48:01.850833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.850805    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"
Oct 29 22:48:01.854624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.854589    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72" event={"ID":"2c8afc25-6b5d-4136-8b69-0ef94a9ae866","Type":"ContainerStarted","Data":"94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086"}
Oct 29 22:48:01.854741 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.854645    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72" event={"ID":"2c8afc25-6b5d-4136-8b69-0ef94a9ae866","Type":"ContainerStarted","Data":"75ddc4f9a0f65913a38b260ed246ea76232d3e7a9919df5ba1601858b1c9449a"}
Oct 29 22:48:01.854741 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.854675    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"
Oct 29 22:48:01.870800 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:48:01.870747    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a7b7f8_7f10_445f_8e90_af03ec8f8498.slice/crio-52d1bb67e19c93e80a8cfcb7dfd93dfe9cfdbf9781ae730c1dec626b04d4907d WatchSource:0}: Error finding container 52d1bb67e19c93e80a8cfcb7dfd93dfe9cfdbf9781ae730c1dec626b04d4907d: Status 404 returned error can't find the container with id 52d1bb67e19c93e80a8cfcb7dfd93dfe9cfdbf9781ae730c1dec626b04d4907d
Oct 29 22:48:01.872443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.872423    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"]
Oct 29 22:48:01.897786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.897745    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m" podStartSLOduration=1.897733531 podStartE2EDuration="1.897733531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:01.892201432 +0000 UTC m=+2023.916977575" watchObservedRunningTime="2026-10-29 22:48:01.897733531 +0000 UTC m=+2023.922509662"
Oct 29 22:48:01.938058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:01.938005    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72" podStartSLOduration=1.93799098 podStartE2EDuration="1.93799098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:01.9329463 +0000 UTC m=+2023.957722432" watchObservedRunningTime="2026-10-29 22:48:01.93799098 +0000 UTC m=+2023.962767115"
Oct 29 22:48:02.865776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:02.865745    4860 generic.go:356] "Generic (PLEG): container finished" podID="40a7b7f8-7f10-445f-8e90-af03ec8f8498" containerID="c8f0f363e3a65b7b4c86079b84c438e86282ac1556be007adf4a18c36c3d4ea7" exitCode=0
Oct 29 22:48:02.866133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:02.865828    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc" event={"ID":"40a7b7f8-7f10-445f-8e90-af03ec8f8498","Type":"ContainerDied","Data":"c8f0f363e3a65b7b4c86079b84c438e86282ac1556be007adf4a18c36c3d4ea7"}
Oct 29 22:48:02.866133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:02.865879    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc" event={"ID":"40a7b7f8-7f10-445f-8e90-af03ec8f8498","Type":"ContainerStarted","Data":"52d1bb67e19c93e80a8cfcb7dfd93dfe9cfdbf9781ae730c1dec626b04d4907d"}
Oct 29 22:48:03.361967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:03.361932    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:04.071800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.071775    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:04.159981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.159893    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-xks-ca\") pod \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\" (UID: \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\") "
Oct 29 22:48:04.159981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.159966    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a7b7f8-7f10-445f-8e90-af03ec8f8498-kube-api-access-stn85\" (UniqueName: \"kubernetes.io/projected/40a7b7f8-7f10-445f-8e90-af03ec8f8498-kube-api-access-stn85\") pod \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\" (UID: \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\") "
Oct 29 22:48:04.160198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.160131    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-openshift-ca\") pod \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\" (UID: \"40a7b7f8-7f10-445f-8e90-af03ec8f8498\") "
Oct 29 22:48:04.161185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.161152    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-openshift-ca" pod "40a7b7f8-7f10-445f-8e90-af03ec8f8498" (UID: "40a7b7f8-7f10-445f-8e90-af03ec8f8498"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:04.173751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.164706    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-xks-ca" pod "40a7b7f8-7f10-445f-8e90-af03ec8f8498" (UID: "40a7b7f8-7f10-445f-8e90-af03ec8f8498"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:04.175103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.175077    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a7b7f8-7f10-445f-8e90-af03ec8f8498-kube-api-access-stn85" pod "40a7b7f8-7f10-445f-8e90-af03ec8f8498" (UID: "40a7b7f8-7f10-445f-8e90-af03ec8f8498"). InnerVolumeSpecName "kube-api-access-stn85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:04.262270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.262241    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:04.262270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.262266    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stn85\" (UniqueName: \"kubernetes.io/projected/40a7b7f8-7f10-445f-8e90-af03ec8f8498-kube-api-access-stn85\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:04.262461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.262276    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40a7b7f8-7f10-445f-8e90-af03ec8f8498-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:04.788805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.788773    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"]
Oct 29 22:48:04.789586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.789569    4860 cpu_manager.go:414] "containerMap: removing container" podUID="40a7b7f8-7f10-445f-8e90-af03ec8f8498" containerName="revoke-keys"
Oct 29 22:48:04.789647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.789591    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a7b7f8-7f10-445f-8e90-af03ec8f8498" containerName="revoke-keys"
Oct 29 22:48:04.789753 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.789743    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a7b7f8-7f10-445f-8e90-af03ec8f8498" containerName="revoke-keys"
Oct 29 22:48:04.793823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.793799    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:04.823893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.823865    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"]
Oct 29 22:48:04.871369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.871318    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b819be94-485nj\" (UID: \"d8dfa7b8-e581-43df-9851-0e57408b78c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:04.871573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.871553    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296st\" (UniqueName: \"kubernetes.io/projected/d8dfa7b8-e581-43df-9851-0e57408b78c1-kube-api-access-296st\") pod \"maas-api-revoke-sub-b819be94-485nj\" (UID: \"d8dfa7b8-e581-43df-9851-0e57408b78c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:04.871752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.871726    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b819be94-485nj\" (UID: \"d8dfa7b8-e581-43df-9851-0e57408b78c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:04.885729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.885703    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"
Oct 29 22:48:04.885866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.885734    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc" event={"ID":"40a7b7f8-7f10-445f-8e90-af03ec8f8498","Type":"ContainerDied","Data":"52d1bb67e19c93e80a8cfcb7dfd93dfe9cfdbf9781ae730c1dec626b04d4907d"}
Oct 29 22:48:04.885866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.885780    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52d1bb67e19c93e80a8cfcb7dfd93dfe9cfdbf9781ae730c1dec626b04d4907d"
Oct 29 22:48:04.973907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.973875    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b819be94-485nj\" (UID: \"d8dfa7b8-e581-43df-9851-0e57408b78c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:04.974048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.974036    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296st\" (UniqueName: \"kubernetes.io/projected/d8dfa7b8-e581-43df-9851-0e57408b78c1-kube-api-access-296st\") pod \"maas-api-revoke-sub-b819be94-485nj\" (UID: \"d8dfa7b8-e581-43df-9851-0e57408b78c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:04.974135 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.974122    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b819be94-485nj\" (UID: \"d8dfa7b8-e581-43df-9851-0e57408b78c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:04.974793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.974775    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b819be94-485nj\" (UID: \"d8dfa7b8-e581-43df-9851-0e57408b78c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:04.976301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.976283    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b819be94-485nj\" (UID: \"d8dfa7b8-e581-43df-9851-0e57408b78c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:04.994670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:04.994652    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-296st\" (UniqueName: \"kubernetes.io/projected/d8dfa7b8-e581-43df-9851-0e57408b78c1-kube-api-access-296st\") pod \"maas-api-revoke-sub-b819be94-485nj\" (UID: \"d8dfa7b8-e581-43df-9851-0e57408b78c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:05.053598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:05.053511    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:05.122350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:05.122057    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:05.450506 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:48:05.450470    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8dfa7b8_e581_43df_9851_0e57408b78c1.slice/crio-14a107e7ab8d57090df1daa7bf2cd740033b6893ddc43879e93fd4226ba9826e WatchSource:0}: Error finding container 14a107e7ab8d57090df1daa7bf2cd740033b6893ddc43879e93fd4226ba9826e: Status 404 returned error can't find the container with id 14a107e7ab8d57090df1daa7bf2cd740033b6893ddc43879e93fd4226ba9826e
Oct 29 22:48:05.453649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:05.453621    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"]
Oct 29 22:48:05.895949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:05.895913    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj" event={"ID":"d8dfa7b8-e581-43df-9851-0e57408b78c1","Type":"ContainerStarted","Data":"14a107e7ab8d57090df1daa7bf2cd740033b6893ddc43879e93fd4226ba9826e"}
Oct 29 22:48:06.553414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.553380    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"]
Oct 29 22:48:06.558945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.558925    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:06.577116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.577087    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"]
Oct 29 22:48:06.707260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.707222    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzbj\" (UniqueName: \"kubernetes.io/projected/33ae10b2-3f73-4b33-a431-069ef827f0f8-kube-api-access-6hzbj\") pod \"maas-api-revoke-sub-309f2427-kzvk8\" (UID: \"33ae10b2-3f73-4b33-a431-069ef827f0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:06.710476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.710454    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-309f2427-kzvk8\" (UID: \"33ae10b2-3f73-4b33-a431-069ef827f0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:06.710625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.710599    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-309f2427-kzvk8\" (UID: \"33ae10b2-3f73-4b33-a431-069ef827f0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:06.727778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.727752    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:48:06.750382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.750359    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:48:06.811308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.811275    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-envoy\") pod \"0357f513-e857-424f-91f8-130b056ca135\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") "
Oct 29 22:48:06.811489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.811318    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-istio-token\" (UniqueName: \"kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-istio-token\") pod \"0357f513-e857-424f-91f8-130b056ca135\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") "
Oct 29 22:48:06.811489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.811445    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-certs\") pod \"0357f513-e857-424f-91f8-130b056ca135\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") "
Oct 29 22:48:06.811489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.811482    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-data\") pod \"0357f513-e857-424f-91f8-130b056ca135\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") "
Oct 29 22:48:06.811646 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.811619    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0357f513-e857-424f-91f8-130b056ca135-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0357f513-e857-424f-91f8-130b056ca135-istio-podinfo\") pod \"0357f513-e857-424f-91f8-130b056ca135\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") "
Oct 29 22:48:06.811755 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.811740    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-credential-socket\") pod \"0357f513-e857-424f-91f8-130b056ca135\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") "
Oct 29 22:48:06.811819 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.811779    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-kube-api-access-d2c8v\" (UniqueName: \"kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-kube-api-access-d2c8v\") pod \"0357f513-e857-424f-91f8-130b056ca135\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") "
Oct 29 22:48:06.811874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.811851    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0357f513-e857-424f-91f8-130b056ca135-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0357f513-e857-424f-91f8-130b056ca135-istiod-ca-cert\") pod \"0357f513-e857-424f-91f8-130b056ca135\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") "
Oct 29 22:48:06.811874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.811851    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-certs" pod "0357f513-e857-424f-91f8-130b056ca135" (UID: "0357f513-e857-424f-91f8-130b056ca135"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:06.812013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.811999    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-socket\") pod \"0357f513-e857-424f-91f8-130b056ca135\" (UID: \"0357f513-e857-424f-91f8-130b056ca135\") "
Oct 29 22:48:06.812151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.812127    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-credential-socket" pod "0357f513-e857-424f-91f8-130b056ca135" (UID: "0357f513-e857-424f-91f8-130b056ca135"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:06.812436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.812274    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0357f513-e857-424f-91f8-130b056ca135-istiod-ca-cert" pod "0357f513-e857-424f-91f8-130b056ca135" (UID: "0357f513-e857-424f-91f8-130b056ca135"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:06.812436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.812380    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-data" pod "0357f513-e857-424f-91f8-130b056ca135" (UID: "0357f513-e857-424f-91f8-130b056ca135"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:06.812436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.812376    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzbj\" (UniqueName: \"kubernetes.io/projected/33ae10b2-3f73-4b33-a431-069ef827f0f8-kube-api-access-6hzbj\") pod \"maas-api-revoke-sub-309f2427-kzvk8\" (UID: \"33ae10b2-3f73-4b33-a431-069ef827f0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:06.812705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.812587    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-socket" pod "0357f513-e857-424f-91f8-130b056ca135" (UID: "0357f513-e857-424f-91f8-130b056ca135"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:06.812705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.812593    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-309f2427-kzvk8\" (UID: \"33ae10b2-3f73-4b33-a431-069ef827f0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:06.812810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.812739    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-309f2427-kzvk8\" (UID: \"33ae10b2-3f73-4b33-a431-069ef827f0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:06.813119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.813094    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.813240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.813123    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0357f513-e857-424f-91f8-130b056ca135-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.813240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.813139    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.813240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.813155    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.813240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.813168    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.813736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.813691    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-309f2427-kzvk8\" (UID: \"33ae10b2-3f73-4b33-a431-069ef827f0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:06.814526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.814404    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-envoy" pod "0357f513-e857-424f-91f8-130b056ca135" (UID: "0357f513-e857-424f-91f8-130b056ca135"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:06.814526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.814495    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-istio-token" pod "0357f513-e857-424f-91f8-130b056ca135" (UID: "0357f513-e857-424f-91f8-130b056ca135"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:06.815241 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.815222    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-309f2427-kzvk8\" (UID: \"33ae10b2-3f73-4b33-a431-069ef827f0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:06.815812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.815778    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-kube-api-access-d2c8v" pod "0357f513-e857-424f-91f8-130b056ca135" (UID: "0357f513-e857-424f-91f8-130b056ca135"). InnerVolumeSpecName "kube-api-access-d2c8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:06.815927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.815818    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0357f513-e857-424f-91f8-130b056ca135-istio-podinfo" pod "0357f513-e857-424f-91f8-130b056ca135" (UID: "0357f513-e857-424f-91f8-130b056ca135"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:48:06.834688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.834660    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzbj\" (UniqueName: \"kubernetes.io/projected/33ae10b2-3f73-4b33-a431-069ef827f0f8-kube-api-access-6hzbj\") pod \"maas-api-revoke-sub-309f2427-kzvk8\" (UID: \"33ae10b2-3f73-4b33-a431-069ef827f0f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:06.842581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.842539    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:06.872964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.872941    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"]
Oct 29 22:48:06.873760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.873744    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0357f513-e857-424f-91f8-130b056ca135" containerName="istio-proxy"
Oct 29 22:48:06.873760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.873761    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0357f513-e857-424f-91f8-130b056ca135" containerName="istio-proxy"
Oct 29 22:48:06.873850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.873775    4860 cpu_manager.go:414] "containerMap: removing container" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerName="istio-proxy"
Oct 29 22:48:06.873850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.873781    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerName="istio-proxy"
Oct 29 22:48:06.873927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.873912    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0357f513-e857-424f-91f8-130b056ca135" containerName="istio-proxy"
Oct 29 22:48:06.873965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.873931    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerName="istio-proxy"
Oct 29 22:48:06.878907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.878890    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:06.896819 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.896794    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"]
Oct 29 22:48:06.907767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.907743    4860 generic.go:356] "Generic (PLEG): container finished" podID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" containerID="dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee" exitCode=0
Oct 29 22:48:06.908101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.908088    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"
Oct 29 22:48:06.908259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.908245    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" event={"ID":"27d54a31-5af8-4540-bb0d-36eaa9efa86f","Type":"ContainerDied","Data":"dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee"}
Oct 29 22:48:06.908303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.908275    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm" event={"ID":"27d54a31-5af8-4540-bb0d-36eaa9efa86f","Type":"ContainerDied","Data":"1f8bcad83de8ca9a6fedb2b527a318fbc7a2955dc7068549eeb118acde5c7da3"}
Oct 29 22:48:06.908303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.908295    4860 scope.go:122] "RemoveContainer" containerID="dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee"
Oct 29 22:48:06.910312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.910275    4860 generic.go:356] "Generic (PLEG): container finished" podID="0357f513-e857-424f-91f8-130b056ca135" containerID="e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30" exitCode=0
Oct 29 22:48:06.910433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.910336    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" event={"ID":"0357f513-e857-424f-91f8-130b056ca135","Type":"ContainerDied","Data":"e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30"}
Oct 29 22:48:06.910433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.910397    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r" event={"ID":"0357f513-e857-424f-91f8-130b056ca135","Type":"ContainerDied","Data":"a695d8e3af79d67f89ce2479e734388ab5fd746eac24c98a927f410e72eac768"}
Oct 29 22:48:06.910433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.910342    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"
Oct 29 22:48:06.912751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.912726    4860 generic.go:356] "Generic (PLEG): container finished" podID="d8dfa7b8-e581-43df-9851-0e57408b78c1" containerID="67da4a43a901d432493cadd867950ef7b888d8b745c54521cf8c1fdd31464992" exitCode=0
Oct 29 22:48:06.912874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.912780    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj" event={"ID":"d8dfa7b8-e581-43df-9851-0e57408b78c1","Type":"ContainerDied","Data":"67da4a43a901d432493cadd867950ef7b888d8b745c54521cf8c1fdd31464992"}
Oct 29 22:48:06.913826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.913759    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-certs\") pod \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") "
Oct 29 22:48:06.913939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.913877    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-kube-api-access-9jn88\" (UniqueName: \"kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-kube-api-access-9jn88\") pod \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") "
Oct 29 22:48:06.913939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.913923    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-credential-socket\") pod \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") "
Oct 29 22:48:06.914073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.914006    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-socket\") pod \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") "
Oct 29 22:48:06.914073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.914066    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-podinfo\") pod \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") "
Oct 29 22:48:06.914189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.914099    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istiod-ca-cert\") pod \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") "
Oct 29 22:48:06.914189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.914141    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-envoy\") pod \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") "
Oct 29 22:48:06.914385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.914221    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-token\" (UniqueName: \"kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-token\") pod \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") "
Oct 29 22:48:06.914385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.914305    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-data\") pod \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\" (UID: \"27d54a31-5af8-4540-bb0d-36eaa9efa86f\") "
Oct 29 22:48:06.914613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.914444    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-certs" pod "27d54a31-5af8-4540-bb0d-36eaa9efa86f" (UID: "27d54a31-5af8-4540-bb0d-36eaa9efa86f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:06.914800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.914777    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-socket" pod "27d54a31-5af8-4540-bb0d-36eaa9efa86f" (UID: "27d54a31-5af8-4540-bb0d-36eaa9efa86f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:06.914800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.914790    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-credential-socket" pod "27d54a31-5af8-4540-bb0d-36eaa9efa86f" (UID: "27d54a31-5af8-4540-bb0d-36eaa9efa86f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:06.915039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.915015    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-data" pod "27d54a31-5af8-4540-bb0d-36eaa9efa86f" (UID: "27d54a31-5af8-4540-bb0d-36eaa9efa86f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:06.915136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.915034    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istiod-ca-cert" pod "27d54a31-5af8-4540-bb0d-36eaa9efa86f" (UID: "27d54a31-5af8-4540-bb0d-36eaa9efa86f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:06.915423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.915401    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0357f513-e857-424f-91f8-130b056ca135-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.915504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.915428    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.915504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.915441    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.915504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.915454    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.915504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.915468    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0357f513-e857-424f-91f8-130b056ca135-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.915504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.915481    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.915504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.915493    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.915839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.915507    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2c8v\" (UniqueName: \"kubernetes.io/projected/0357f513-e857-424f-91f8-130b056ca135-kube-api-access-d2c8v\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.915839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.915520    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:06.916933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.916908    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-kube-api-access-9jn88" pod "27d54a31-5af8-4540-bb0d-36eaa9efa86f" (UID: "27d54a31-5af8-4540-bb0d-36eaa9efa86f"). InnerVolumeSpecName "kube-api-access-9jn88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:06.921943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.919665    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-podinfo" pod "27d54a31-5af8-4540-bb0d-36eaa9efa86f" (UID: "27d54a31-5af8-4540-bb0d-36eaa9efa86f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:48:06.921943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.919710    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-envoy" pod "27d54a31-5af8-4540-bb0d-36eaa9efa86f" (UID: "27d54a31-5af8-4540-bb0d-36eaa9efa86f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:06.921943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.921097    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-token" pod "27d54a31-5af8-4540-bb0d-36eaa9efa86f" (UID: "27d54a31-5af8-4540-bb0d-36eaa9efa86f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:06.925487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.925469    4860 scope.go:122] "RemoveContainer" containerID="dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee"
Oct 29 22:48:06.925773 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:48:06.925752    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee\": container with ID starting with dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee not found: ID does not exist" containerID="dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee"
Oct 29 22:48:06.925857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.925782    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee"} err="failed to get container status \"dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee\": rpc error: code = NotFound desc = could not find container \"dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee\": container with ID starting with dc4b5a404efd1625e2005c7cbd4bcfafe7c9089d865d56b1de83417b9adb21ee not found: ID does not exist"
Oct 29 22:48:06.925857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.925807    4860 scope.go:122] "RemoveContainer" containerID="e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30"
Oct 29 22:48:06.955950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.954601    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:06.987594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.987510    4860 scope.go:122] "RemoveContainer" containerID="e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30"
Oct 29 22:48:06.987951 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:48:06.987922    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30\": container with ID starting with e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30 not found: ID does not exist" containerID="e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30"
Oct 29 22:48:06.988100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:06.987961    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30"} err="failed to get container status \"e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30\": rpc error: code = NotFound desc = could not find container \"e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30\": container with ID starting with e45782d1b859c5316577e02f11e141ce67e75ee51f73a4880c6dd6d5955a3e30 not found: ID does not exist"
Oct 29 22:48:07.017510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.017471    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldmb\" (UniqueName: \"kubernetes.io/projected/c0551700-3fdf-4bee-b534-0bd0a66278dc-kube-api-access-jldmb\") pod \"maas-api-revoke-sub-406a9910-blsvx\" (UID: \"c0551700-3fdf-4bee-b534-0bd0a66278dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:07.017716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.017696    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406a9910-blsvx\" (UID: \"c0551700-3fdf-4bee-b534-0bd0a66278dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:07.017794 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.017774    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406a9910-blsvx\" (UID: \"c0551700-3fdf-4bee-b534-0bd0a66278dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:07.019126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.019100    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jn88\" (UniqueName: \"kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-kube-api-access-9jn88\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:07.019274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.019130    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:07.019274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.019146    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:07.019274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.019159    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27d54a31-5af8-4540-bb0d-36eaa9efa86f-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:07.090383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.090310    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"]
Oct 29 22:48:07.098214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.098190    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-82d8r"]
Oct 29 22:48:07.120283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.120249    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406a9910-blsvx\" (UID: \"c0551700-3fdf-4bee-b534-0bd0a66278dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:07.120541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.120519    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406a9910-blsvx\" (UID: \"c0551700-3fdf-4bee-b534-0bd0a66278dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:07.120856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.120836    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldmb\" (UniqueName: \"kubernetes.io/projected/c0551700-3fdf-4bee-b534-0bd0a66278dc-kube-api-access-jldmb\") pod \"maas-api-revoke-sub-406a9910-blsvx\" (UID: \"c0551700-3fdf-4bee-b534-0bd0a66278dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:07.122721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.122697    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406a9910-blsvx\" (UID: \"c0551700-3fdf-4bee-b534-0bd0a66278dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:07.128745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.128719    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406a9910-blsvx\" (UID: \"c0551700-3fdf-4bee-b534-0bd0a66278dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:07.141803 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.141783    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldmb\" (UniqueName: \"kubernetes.io/projected/c0551700-3fdf-4bee-b534-0bd0a66278dc-kube-api-access-jldmb\") pod \"maas-api-revoke-sub-406a9910-blsvx\" (UID: \"c0551700-3fdf-4bee-b534-0bd0a66278dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:07.194985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.194959    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:07.266086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.266043    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"]
Oct 29 22:48:07.271658 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.271629    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-xvdlm"]
Oct 29 22:48:07.296389 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:48:07.296362    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ae10b2_3f73_4b33_a431_069ef827f0f8.slice/crio-b16481548ca51852e65a5cea82a024c18b18a3600ca27776f8e15f0b06c1e833 WatchSource:0}: Error finding container b16481548ca51852e65a5cea82a024c18b18a3600ca27776f8e15f0b06c1e833: Status 404 returned error can't find the container with id b16481548ca51852e65a5cea82a024c18b18a3600ca27776f8e15f0b06c1e833
Oct 29 22:48:07.298062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.298026    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"]
Oct 29 22:48:07.498948 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:48:07.498918    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0551700_3fdf_4bee_b534_0bd0a66278dc.slice/crio-453029425a70bc546c98a135ae2a67b9de2d3e6378ca62c064671767796a9281 WatchSource:0}: Error finding container 453029425a70bc546c98a135ae2a67b9de2d3e6378ca62c064671767796a9281: Status 404 returned error can't find the container with id 453029425a70bc546c98a135ae2a67b9de2d3e6378ca62c064671767796a9281
Oct 29 22:48:07.501934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.501909    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"]
Oct 29 22:48:07.924242 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.924215    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx" event={"ID":"c0551700-3fdf-4bee-b534-0bd0a66278dc","Type":"ContainerStarted","Data":"ee1195a47a4094b0d5ba253c8c0bb0c91afdb210d35a589620b59260c37f091b"}
Oct 29 22:48:07.924642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.924265    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx" event={"ID":"c0551700-3fdf-4bee-b534-0bd0a66278dc","Type":"ContainerStarted","Data":"453029425a70bc546c98a135ae2a67b9de2d3e6378ca62c064671767796a9281"}
Oct 29 22:48:07.927061 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.927035    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8" event={"ID":"33ae10b2-3f73-4b33-a431-069ef827f0f8","Type":"ContainerStarted","Data":"bda4345e104cc50b8d72d9a3caec01feb72fd764973d395c5ed6dceba8845162"}
Oct 29 22:48:07.927150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.927084    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8" event={"ID":"33ae10b2-3f73-4b33-a431-069ef827f0f8","Type":"ContainerStarted","Data":"b16481548ca51852e65a5cea82a024c18b18a3600ca27776f8e15f0b06c1e833"}
Oct 29 22:48:07.974176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:07.974116    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx" podStartSLOduration=1.974098326 podStartE2EDuration="1.974098326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:07.967407596 +0000 UTC m=+2029.992183754" watchObservedRunningTime="2026-10-29 22:48:07.974098326 +0000 UTC m=+2029.998874464"
Oct 29 22:48:08.013953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.013890    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8" podStartSLOduration=2.013868582 podStartE2EDuration="2.013868582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:08.009608675 +0000 UTC m=+2030.034384833" watchObservedRunningTime="2026-10-29 22:48:08.013868582 +0000 UTC m=+2030.038644714"
Oct 29 22:48:08.109762 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.109733    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:08.115736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.115711    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0357f513-e857-424f-91f8-130b056ca135" path="/var/lib/kubelet/pods/0357f513-e857-424f-91f8-130b056ca135/volumes"
Oct 29 22:48:08.116284 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.116270    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d54a31-5af8-4540-bb0d-36eaa9efa86f" path="/var/lib/kubelet/pods/27d54a31-5af8-4540-bb0d-36eaa9efa86f/volumes"
Oct 29 22:48:08.238998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.238966    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-openshift-ca\") pod \"d8dfa7b8-e581-43df-9851-0e57408b78c1\" (UID: \"d8dfa7b8-e581-43df-9851-0e57408b78c1\") "
Oct 29 22:48:08.239169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.239032    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-xks-ca\") pod \"d8dfa7b8-e581-43df-9851-0e57408b78c1\" (UID: \"d8dfa7b8-e581-43df-9851-0e57408b78c1\") "
Oct 29 22:48:08.239169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.239089    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8dfa7b8-e581-43df-9851-0e57408b78c1-kube-api-access-296st\" (UniqueName: \"kubernetes.io/projected/d8dfa7b8-e581-43df-9851-0e57408b78c1-kube-api-access-296st\") pod \"d8dfa7b8-e581-43df-9851-0e57408b78c1\" (UID: \"d8dfa7b8-e581-43df-9851-0e57408b78c1\") "
Oct 29 22:48:08.239404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.239372    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-openshift-ca" pod "d8dfa7b8-e581-43df-9851-0e57408b78c1" (UID: "d8dfa7b8-e581-43df-9851-0e57408b78c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:08.239777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.239756    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:08.241234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.241205    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-xks-ca" pod "d8dfa7b8-e581-43df-9851-0e57408b78c1" (UID: "d8dfa7b8-e581-43df-9851-0e57408b78c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:08.241311 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.241249    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8dfa7b8-e581-43df-9851-0e57408b78c1-kube-api-access-296st" pod "d8dfa7b8-e581-43df-9851-0e57408b78c1" (UID: "d8dfa7b8-e581-43df-9851-0e57408b78c1"). InnerVolumeSpecName "kube-api-access-296st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:08.341173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.341114    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8dfa7b8-e581-43df-9851-0e57408b78c1-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:08.341173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.341140    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-296st\" (UniqueName: \"kubernetes.io/projected/d8dfa7b8-e581-43df-9851-0e57408b78c1-kube-api-access-296st\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:08.465257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.465222    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:08.937434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.937397    4860 generic.go:356] "Generic (PLEG): container finished" podID="33ae10b2-3f73-4b33-a431-069ef827f0f8" containerID="bda4345e104cc50b8d72d9a3caec01feb72fd764973d395c5ed6dceba8845162" exitCode=0
Oct 29 22:48:08.937864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.937522    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8" event={"ID":"33ae10b2-3f73-4b33-a431-069ef827f0f8","Type":"ContainerDied","Data":"bda4345e104cc50b8d72d9a3caec01feb72fd764973d395c5ed6dceba8845162"}
Oct 29 22:48:08.939159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.939136    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"]
Oct 29 22:48:08.939302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.939277    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj" event={"ID":"d8dfa7b8-e581-43df-9851-0e57408b78c1","Type":"ContainerDied","Data":"14a107e7ab8d57090df1daa7bf2cd740033b6893ddc43879e93fd4226ba9826e"}
Oct 29 22:48:08.939380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.939304    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"
Oct 29 22:48:08.939380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.939340    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14a107e7ab8d57090df1daa7bf2cd740033b6893ddc43879e93fd4226ba9826e"
Oct 29 22:48:08.939380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.939354    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" podUID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" containerName="istio-proxy" containerID="cri-o://419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec" gracePeriod=30
Oct 29 22:48:08.941134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.941110    4860 generic.go:356] "Generic (PLEG): container finished" podID="c0551700-3fdf-4bee-b534-0bd0a66278dc" containerID="ee1195a47a4094b0d5ba253c8c0bb0c91afdb210d35a589620b59260c37f091b" exitCode=0
Oct 29 22:48:08.941220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:08.941154    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx" event={"ID":"c0551700-3fdf-4bee-b534-0bd0a66278dc","Type":"ContainerDied","Data":"ee1195a47a4094b0d5ba253c8c0bb0c91afdb210d35a589620b59260c37f091b"}
Oct 29 22:48:09.992268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:09.992226    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:10.216988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.216930    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:10.219874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.219852    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:10.384855 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.384826    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0551700-3fdf-4bee-b534-0bd0a66278dc-kube-api-access-jldmb\" (UniqueName: \"kubernetes.io/projected/c0551700-3fdf-4bee-b534-0bd0a66278dc-kube-api-access-jldmb\") pod \"c0551700-3fdf-4bee-b534-0bd0a66278dc\" (UID: \"c0551700-3fdf-4bee-b534-0bd0a66278dc\") "
Oct 29 22:48:10.385028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.384864    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ae10b2-3f73-4b33-a431-069ef827f0f8-kube-api-access-6hzbj\" (UniqueName: \"kubernetes.io/projected/33ae10b2-3f73-4b33-a431-069ef827f0f8-kube-api-access-6hzbj\") pod \"33ae10b2-3f73-4b33-a431-069ef827f0f8\" (UID: \"33ae10b2-3f73-4b33-a431-069ef827f0f8\") "
Oct 29 22:48:10.385028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.384940    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-openshift-ca\") pod \"33ae10b2-3f73-4b33-a431-069ef827f0f8\" (UID: \"33ae10b2-3f73-4b33-a431-069ef827f0f8\") "
Oct 29 22:48:10.385028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.384965    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-openshift-ca\") pod \"c0551700-3fdf-4bee-b534-0bd0a66278dc\" (UID: \"c0551700-3fdf-4bee-b534-0bd0a66278dc\") "
Oct 29 22:48:10.385028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.385003    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-xks-ca\") pod \"c0551700-3fdf-4bee-b534-0bd0a66278dc\" (UID: \"c0551700-3fdf-4bee-b534-0bd0a66278dc\") "
Oct 29 22:48:10.385251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.385049    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-xks-ca\") pod \"33ae10b2-3f73-4b33-a431-069ef827f0f8\" (UID: \"33ae10b2-3f73-4b33-a431-069ef827f0f8\") "
Oct 29 22:48:10.385308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.385278    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-openshift-ca" pod "33ae10b2-3f73-4b33-a431-069ef827f0f8" (UID: "33ae10b2-3f73-4b33-a431-069ef827f0f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:10.385386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.385356    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-openshift-ca" pod "c0551700-3fdf-4bee-b534-0bd0a66278dc" (UID: "c0551700-3fdf-4bee-b534-0bd0a66278dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:10.385922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.385899    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:10.385922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.385921    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:10.387403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.387295    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-xks-ca" pod "33ae10b2-3f73-4b33-a431-069ef827f0f8" (UID: "33ae10b2-3f73-4b33-a431-069ef827f0f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:10.387403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.387341    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ae10b2-3f73-4b33-a431-069ef827f0f8-kube-api-access-6hzbj" pod "33ae10b2-3f73-4b33-a431-069ef827f0f8" (UID: "33ae10b2-3f73-4b33-a431-069ef827f0f8"). InnerVolumeSpecName "kube-api-access-6hzbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:10.387691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.387665    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-xks-ca" pod "c0551700-3fdf-4bee-b534-0bd0a66278dc" (UID: "c0551700-3fdf-4bee-b534-0bd0a66278dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:10.387691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.387688    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0551700-3fdf-4bee-b534-0bd0a66278dc-kube-api-access-jldmb" pod "c0551700-3fdf-4bee-b534-0bd0a66278dc" (UID: "c0551700-3fdf-4bee-b534-0bd0a66278dc"). InnerVolumeSpecName "kube-api-access-jldmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:10.487049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.486992    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jldmb\" (UniqueName: \"kubernetes.io/projected/c0551700-3fdf-4bee-b534-0bd0a66278dc-kube-api-access-jldmb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:10.487049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.487022    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hzbj\" (UniqueName: \"kubernetes.io/projected/33ae10b2-3f73-4b33-a431-069ef827f0f8-kube-api-access-6hzbj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:10.487049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.487034    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0551700-3fdf-4bee-b534-0bd0a66278dc-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:10.487049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.487043    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ae10b2-3f73-4b33-a431-069ef827f0f8-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:10.960959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.960931    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"
Oct 29 22:48:10.961148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.960953    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx" event={"ID":"c0551700-3fdf-4bee-b534-0bd0a66278dc","Type":"ContainerDied","Data":"453029425a70bc546c98a135ae2a67b9de2d3e6378ca62c064671767796a9281"}
Oct 29 22:48:10.961148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.961006    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="453029425a70bc546c98a135ae2a67b9de2d3e6378ca62c064671767796a9281"
Oct 29 22:48:10.962777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.962750    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"
Oct 29 22:48:10.962923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.962793    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8" event={"ID":"33ae10b2-3f73-4b33-a431-069ef827f0f8","Type":"ContainerDied","Data":"b16481548ca51852e65a5cea82a024c18b18a3600ca27776f8e15f0b06c1e833"}
Oct 29 22:48:10.962923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:10.962830    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b16481548ca51852e65a5cea82a024c18b18a3600ca27776f8e15f0b06c1e833"
Oct 29 22:48:11.058052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:11.058018    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:11.172247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:11.172214    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:11.555032 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:11.554995    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"]
Oct 29 22:48:11.597913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:11.597420    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"]
Oct 29 22:48:11.661003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:11.660967    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:11.774600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:11.774568    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:12.869624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:12.869596    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"
Oct 29 22:48:12.870034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:12.869965    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"
Oct 29 22:48:13.311598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:13.311561    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:13.438513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:13.438476    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:14.236176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.236151    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:48:14.266846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.266820    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-socket\") pod \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") "
Oct 29 22:48:14.266968 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.266902    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-kube-api-access-wvmln\" (UniqueName: \"kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-kube-api-access-wvmln\") pod \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") "
Oct 29 22:48:14.266968 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.266936    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istiod-ca-cert\") pod \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") "
Oct 29 22:48:14.267063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.267044    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-envoy\") pod \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") "
Oct 29 22:48:14.267112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.267095    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-socket" pod "ad3ea6e5-4dea-45ea-905e-b8091d29b209" (UID: "ad3ea6e5-4dea-45ea-905e-b8091d29b209"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:14.267172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.267159    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-data\") pod \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") "
Oct 29 22:48:14.267219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.267206    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-token\" (UniqueName: \"kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-token\") pod \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") "
Oct 29 22:48:14.267275 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.267256    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-credential-socket\") pod \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") "
Oct 29 22:48:14.267360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.267317    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-certs\") pod \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") "
Oct 29 22:48:14.267484 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.267462    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-data" pod "ad3ea6e5-4dea-45ea-905e-b8091d29b209" (UID: "ad3ea6e5-4dea-45ea-905e-b8091d29b209"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:14.267571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.267472    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-podinfo\") pod \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\" (UID: \"ad3ea6e5-4dea-45ea-905e-b8091d29b209\") "
Oct 29 22:48:14.267626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.267572    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-credential-socket" pod "ad3ea6e5-4dea-45ea-905e-b8091d29b209" (UID: "ad3ea6e5-4dea-45ea-905e-b8091d29b209"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:14.267725 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.267696    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-certs" pod "ad3ea6e5-4dea-45ea-905e-b8091d29b209" (UID: "ad3ea6e5-4dea-45ea-905e-b8091d29b209"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:14.267856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.267755    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istiod-ca-cert" pod "ad3ea6e5-4dea-45ea-905e-b8091d29b209" (UID: "ad3ea6e5-4dea-45ea-905e-b8091d29b209"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:14.268203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.268175    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:14.268314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.268207    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:14.268314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.268222    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:14.268314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.268236    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:14.268314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.268250    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:14.269315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.269291    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-envoy" pod "ad3ea6e5-4dea-45ea-905e-b8091d29b209" (UID: "ad3ea6e5-4dea-45ea-905e-b8091d29b209"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:48:14.269443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.269425    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-kube-api-access-wvmln" pod "ad3ea6e5-4dea-45ea-905e-b8091d29b209" (UID: "ad3ea6e5-4dea-45ea-905e-b8091d29b209"). InnerVolumeSpecName "kube-api-access-wvmln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:14.269510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.269451    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-token" pod "ad3ea6e5-4dea-45ea-905e-b8091d29b209" (UID: "ad3ea6e5-4dea-45ea-905e-b8091d29b209"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:14.269675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.269659    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-podinfo" pod "ad3ea6e5-4dea-45ea-905e-b8091d29b209" (UID: "ad3ea6e5-4dea-45ea-905e-b8091d29b209"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:48:14.369075 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.369039    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:14.369075 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.369069    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:14.369319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.369083    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvmln\" (UniqueName: \"kubernetes.io/projected/ad3ea6e5-4dea-45ea-905e-b8091d29b209-kube-api-access-wvmln\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:14.369319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.369095    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3ea6e5-4dea-45ea-905e-b8091d29b209-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:14.930127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:14.930087    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:15.007853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:15.007820    4860 generic.go:356] "Generic (PLEG): container finished" podID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" containerID="419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec" exitCode=0
Oct 29 22:48:15.008062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:15.007871    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"
Oct 29 22:48:15.008062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:15.007908    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" event={"ID":"ad3ea6e5-4dea-45ea-905e-b8091d29b209","Type":"ContainerDied","Data":"419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec"}
Oct 29 22:48:15.008062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:15.007969    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb" event={"ID":"ad3ea6e5-4dea-45ea-905e-b8091d29b209","Type":"ContainerDied","Data":"dafc98cc33fd623604f07d603173c3e0cedc96962d83db75df83384c6cd10087"}
Oct 29 22:48:15.008062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:15.008002    4860 scope.go:122] "RemoveContainer" containerID="419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec"
Oct 29 22:48:15.023736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:15.023717    4860 scope.go:122] "RemoveContainer" containerID="419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec"
Oct 29 22:48:15.024041 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:48:15.024020    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec\": container with ID starting with 419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec not found: ID does not exist" containerID="419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec"
Oct 29 22:48:15.024098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:15.024050    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec"} err="failed to get container status \"419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec\": rpc error: code = NotFound desc = could not find container \"419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec\": container with ID starting with 419e5a3f79d38dfd2fab4f6a193d6fc92f6343f86f0cfcf31def0bec63d5a7ec not found: ID does not exist"
Oct 29 22:48:15.068745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:15.068708    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"]
Oct 29 22:48:15.082072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:15.082050    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-ghbxb"]
Oct 29 22:48:16.114470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:16.114439    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" path="/var/lib/kubelet/pods/ad3ea6e5-4dea-45ea-905e-b8091d29b209/volumes"
Oct 29 22:48:16.185887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:16.185858    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:16.324132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:16.324091    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:16.555521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:16.555485    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx" podUID="70d957f1-c5a8-4993-a3af-2e35b2934cc4" containerName="payload-processing" containerID="cri-o://a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce" gracePeriod=25
Oct 29 22:48:16.598485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:16.598450    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t" podUID="1e35c643-1248-47c8-8acd-1a4991f0a395" containerName="payload-pre-processing" containerID="cri-o://1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652" gracePeriod=25
Oct 29 22:48:16.634881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:16.634836    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:18.060175 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:48:18.060136    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:48:18.241527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.241494    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:18.552991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.552962    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"]
Oct 29 22:48:18.553737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.553723    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d8dfa7b8-e581-43df-9851-0e57408b78c1" containerName="revoke-keys"
Oct 29 22:48:18.553797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.553738    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8dfa7b8-e581-43df-9851-0e57408b78c1" containerName="revoke-keys"
Oct 29 22:48:18.553797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.553756    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" containerName="istio-proxy"
Oct 29 22:48:18.553797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.553761    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" containerName="istio-proxy"
Oct 29 22:48:18.553797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.553776    4860 cpu_manager.go:414] "containerMap: removing container" podUID="33ae10b2-3f73-4b33-a431-069ef827f0f8" containerName="revoke-keys"
Oct 29 22:48:18.553797 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.553781    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ae10b2-3f73-4b33-a431-069ef827f0f8" containerName="revoke-keys"
Oct 29 22:48:18.553961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.553814    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c0551700-3fdf-4bee-b534-0bd0a66278dc" containerName="revoke-keys"
Oct 29 22:48:18.553961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.553818    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0551700-3fdf-4bee-b534-0bd0a66278dc" containerName="revoke-keys"
Oct 29 22:48:18.553961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.553936    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0551700-3fdf-4bee-b534-0bd0a66278dc" containerName="revoke-keys"
Oct 29 22:48:18.553961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.553949    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8dfa7b8-e581-43df-9851-0e57408b78c1" containerName="revoke-keys"
Oct 29 22:48:18.553961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.553956    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3ea6e5-4dea-45ea-905e-b8091d29b209" containerName="istio-proxy"
Oct 29 22:48:18.554120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.553963    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ae10b2-3f73-4b33-a431-069ef827f0f8" containerName="revoke-keys"
Oct 29 22:48:18.559531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.559512    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.582523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.582501    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"]
Oct 29 22:48:18.590379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.590355    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2a3ef0-openshift-default-dockercfg-wm7cg\""
Oct 29 22:48:18.629727 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.629700    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istiod-ca-cert\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.629860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.629831    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-credential-socket\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.629908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.629888    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6kc\" (UniqueName: \"kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-kube-api-access-9d6kc\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.629951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.629919    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-certs\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.630157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.630134    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-podinfo\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.630288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.630271    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-envoy\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.630393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.630376    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-socket\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.630477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.630457    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-data\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.630558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.630527    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-token\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.731392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.731353    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-podinfo\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.731584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.731461    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-envoy\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.731584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.731523    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-socket\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.731584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.731568    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-data\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.731787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.731641    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-token\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.731787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.731673    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istiod-ca-cert\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.731787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.731734    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-credential-socket\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.731787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.731766    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6kc\" (UniqueName: \"kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-kube-api-access-9d6kc\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.732024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.731795    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-certs\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.732024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.731966    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-socket\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.732696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.732308    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-data\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.732696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.732500    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-credential-socket\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.732899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.732867    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istiod-ca-cert\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.733599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.733579    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-certs\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.734104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.734079    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-podinfo\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.734300 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.734276    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-envoy\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.752899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.752871    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6kc\" (UniqueName: \"kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-kube-api-access-9d6kc\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.753018 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.752985    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-token\") pod \"e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") " pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:18.875569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:18.875501    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:19.194519 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:48:19.194489    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3538b7e_90e3_4cbc_8d32_a82d02f0dae1.slice/crio-98e6f7e2e7b3773c1247e814353ec57ec8d485eb6f384cdfd44bf77ddc32f463 WatchSource:0}: Error finding container 98e6f7e2e7b3773c1247e814353ec57ec8d485eb6f384cdfd44bf77ddc32f463: Status 404 returned error can't find the container with id 98e6f7e2e7b3773c1247e814353ec57ec8d485eb6f384cdfd44bf77ddc32f463
Oct 29 22:48:19.198069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.198024    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:48:19.198147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.198123    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:48:19.198198 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.198167    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:48:19.199562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.199542    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"]
Oct 29 22:48:19.452820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.452731    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:19.538225 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.538019    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx" podUID="70d957f1-c5a8-4993-a3af-2e35b2934cc4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:48:19.557388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.541425    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t" podUID="1e35c643-1248-47c8-8acd-1a4991f0a395" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:48:19.947670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.947635    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"]
Oct 29 22:48:19.953627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.953605    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:19.966932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.966872    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:19.969512 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.969489    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"]
Oct 29 22:48:19.976408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.976387    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:48:19.976598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.976550    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:48:19.976695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.976662    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:48:19.985359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:19.985336    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:48:20.056424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.056378    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c716360b-x26bw\" (UID: \"87d833d0-c4c3-4e18-809e-4ec1a1965117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:20.056768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.056516    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c716360b-x26bw\" (UID: \"87d833d0-c4c3-4e18-809e-4ec1a1965117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:20.056768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.056609    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6krf\" (UniqueName: \"kubernetes.io/projected/87d833d0-c4c3-4e18-809e-4ec1a1965117-kube-api-access-q6krf\") pod \"maas-api-revoke-sub-c716360b-x26bw\" (UID: \"87d833d0-c4c3-4e18-809e-4ec1a1965117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:20.067503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.067192    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs" event={"ID":"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1","Type":"ContainerStarted","Data":"6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e"}
Oct 29 22:48:20.067503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.067270    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs" event={"ID":"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1","Type":"ContainerStarted","Data":"98e6f7e2e7b3773c1247e814353ec57ec8d485eb6f384cdfd44bf77ddc32f463"}
Oct 29 22:48:20.113369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.113309    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs" podStartSLOduration=2.113295923 podStartE2EDuration="2.113295923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:20.111124369 +0000 UTC m=+2042.135900501" watchObservedRunningTime="2026-10-29 22:48:20.113295923 +0000 UTC m=+2042.138072054"
Oct 29 22:48:20.158921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.158875    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c716360b-x26bw\" (UID: \"87d833d0-c4c3-4e18-809e-4ec1a1965117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:20.161346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.160036    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c716360b-x26bw\" (UID: \"87d833d0-c4c3-4e18-809e-4ec1a1965117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:20.161346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.160250    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c716360b-x26bw\" (UID: \"87d833d0-c4c3-4e18-809e-4ec1a1965117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:20.161346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.160425    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6krf\" (UniqueName: \"kubernetes.io/projected/87d833d0-c4c3-4e18-809e-4ec1a1965117-kube-api-access-q6krf\") pod \"maas-api-revoke-sub-c716360b-x26bw\" (UID: \"87d833d0-c4c3-4e18-809e-4ec1a1965117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:20.163467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.163443    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c716360b-x26bw\" (UID: \"87d833d0-c4c3-4e18-809e-4ec1a1965117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:20.195852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.195811    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6krf\" (UniqueName: \"kubernetes.io/projected/87d833d0-c4c3-4e18-809e-4ec1a1965117-kube-api-access-q6krf\") pod \"maas-api-revoke-sub-c716360b-x26bw\" (UID: \"87d833d0-c4c3-4e18-809e-4ec1a1965117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:20.347366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.347268    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:20.701031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.701003    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"]
Oct 29 22:48:20.889453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.886662    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:20.892746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.891440    4860 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 29 22:48:20.892746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:20.891512    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs" podUID="b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 29 22:48:21.102379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:21.102295    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw" event={"ID":"87d833d0-c4c3-4e18-809e-4ec1a1965117","Type":"ContainerStarted","Data":"0f0963211dfcc1ae087452e536518c1f9bd5dba7c4bc92902579c8e6b1427467"}
Oct 29 22:48:21.648856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:21.648823    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:21.892709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:21.892672    4860 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 29 22:48:21.892886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:21.892730    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs" podUID="b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 29 22:48:22.113025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:22.112995    4860 generic.go:356] "Generic (PLEG): container finished" podID="87d833d0-c4c3-4e18-809e-4ec1a1965117" containerID="b5bbd62bfc29511df8e8bab1a1c1f773a080a0870136fabf896ac15dc175ce79" exitCode=0
Oct 29 22:48:22.114793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:22.114767    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw" event={"ID":"87d833d0-c4c3-4e18-809e-4ec1a1965117","Type":"ContainerDied","Data":"b5bbd62bfc29511df8e8bab1a1c1f773a080a0870136fabf896ac15dc175ce79"}
Oct 29 22:48:22.892478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:22.892429    4860 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 29 22:48:22.892912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:22.892482    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs" podUID="b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 29 22:48:22.900385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:22.900357    4860 scope.go:122] "RemoveContainer" containerID="cd890ffce2c83796dfc8a0bd27cefe0fac30c655498e99c5c8a3bc62e8601703"
Oct 29 22:48:23.138810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.138762    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:23.436519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.436495    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:23.548575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.548501    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-xks-ca\") pod \"87d833d0-c4c3-4e18-809e-4ec1a1965117\" (UID: \"87d833d0-c4c3-4e18-809e-4ec1a1965117\") "
Oct 29 22:48:23.548719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.548588    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d833d0-c4c3-4e18-809e-4ec1a1965117-kube-api-access-q6krf\" (UniqueName: \"kubernetes.io/projected/87d833d0-c4c3-4e18-809e-4ec1a1965117-kube-api-access-q6krf\") pod \"87d833d0-c4c3-4e18-809e-4ec1a1965117\" (UID: \"87d833d0-c4c3-4e18-809e-4ec1a1965117\") "
Oct 29 22:48:23.548873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.548853    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-openshift-ca\") pod \"87d833d0-c4c3-4e18-809e-4ec1a1965117\" (UID: \"87d833d0-c4c3-4e18-809e-4ec1a1965117\") "
Oct 29 22:48:23.549189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.549159    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-openshift-ca" pod "87d833d0-c4c3-4e18-809e-4ec1a1965117" (UID: "87d833d0-c4c3-4e18-809e-4ec1a1965117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:23.549618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.549595    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:23.550959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.550933    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-xks-ca" pod "87d833d0-c4c3-4e18-809e-4ec1a1965117" (UID: "87d833d0-c4c3-4e18-809e-4ec1a1965117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:23.551056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.551014    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d833d0-c4c3-4e18-809e-4ec1a1965117-kube-api-access-q6krf" pod "87d833d0-c4c3-4e18-809e-4ec1a1965117" (UID: "87d833d0-c4c3-4e18-809e-4ec1a1965117"). InnerVolumeSpecName "kube-api-access-q6krf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:23.651157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.651127    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d833d0-c4c3-4e18-809e-4ec1a1965117-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:23.651157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.651155    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6krf\" (UniqueName: \"kubernetes.io/projected/87d833d0-c4c3-4e18-809e-4ec1a1965117-kube-api-access-q6krf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:23.883409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.879397    4860 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 29 22:48:23.883409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:23.879456    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs" podUID="b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 29 22:48:24.133581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.133517    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"
Oct 29 22:48:24.133581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.133550    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw" event={"ID":"87d833d0-c4c3-4e18-809e-4ec1a1965117","Type":"ContainerDied","Data":"0f0963211dfcc1ae087452e536518c1f9bd5dba7c4bc92902579c8e6b1427467"}
Oct 29 22:48:24.134029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.133595    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f0963211dfcc1ae087452e536518c1f9bd5dba7c4bc92902579c8e6b1427467"
Oct 29 22:48:24.603462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.603424    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"]
Oct 29 22:48:24.604665 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.604644    4860 cpu_manager.go:414] "containerMap: removing container" podUID="87d833d0-c4c3-4e18-809e-4ec1a1965117" containerName="revoke-keys"
Oct 29 22:48:24.604758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.604666    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d833d0-c4c3-4e18-809e-4ec1a1965117" containerName="revoke-keys"
Oct 29 22:48:24.604877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.604864    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d833d0-c4c3-4e18-809e-4ec1a1965117" containerName="revoke-keys"
Oct 29 22:48:24.609648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.609628    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.627212 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.627178    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"]
Oct 29 22:48:24.632376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.632310    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-d68c2c-serving-cert\""
Oct 29 22:48:24.632376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.632339    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:48:24.632566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.632343    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dzwhk\""
Oct 29 22:48:24.632566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.632380    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:48:24.635000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.634971    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 22:48:24.666805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.666775    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.667089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.667065    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2604c6c-436e-4aa2-9ec6-fe306749909d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.667197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.667179    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr7k8\" (UniqueName: \"kubernetes.io/projected/b2604c6c-436e-4aa2-9ec6-fe306749909d-kube-api-access-xr7k8\") pod \"maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.690345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.667474    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.793351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.792432    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.793530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.793485    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.793732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.793703    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2604c6c-436e-4aa2-9ec6-fe306749909d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.793842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.793789    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr7k8\" (UniqueName: \"kubernetes.io/projected/b2604c6c-436e-4aa2-9ec6-fe306749909d-kube-api-access-xr7k8\") pod \"maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.794831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.794798    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2604c6c-436e-4aa2-9ec6-fe306749909d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.795064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.795035    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:24.796518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.796489    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.796795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.796775    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.830509 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.830488    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr7k8\" (UniqueName: \"kubernetes.io/projected/b2604c6c-436e-4aa2-9ec6-fe306749909d-kube-api-access-xr7k8\") pod \"maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.903456 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.897077    4860 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 29 22:48:24.903456 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.897139    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs" podUID="b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 29 22:48:24.927590 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.927557    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:24.954944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:24.954914    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:25.300477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:25.300442    4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:48:25.303297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:25.303274    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"]
Oct 29 22:48:25.876851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:25.876776    4860 patch_prober.go:29] interesting pod/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body=
Oct 29 22:48:25.876851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:25.876837    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs" podUID="b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused"
Oct 29 22:48:26.159679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:26.159594    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5" event={"ID":"b2604c6c-436e-4aa2-9ec6-fe306749909d","Type":"ContainerStarted","Data":"7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7"}
Oct 29 22:48:26.159679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:26.159655    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5" event={"ID":"b2604c6c-436e-4aa2-9ec6-fe306749909d","Type":"ContainerStarted","Data":"c5c1c1f67d795b997ffc932943fd9c80e00ff2eae45dfc3e4f3101f9798bc69d"}
Oct 29 22:48:26.159868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:26.159681    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:26.206172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:26.206101    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5" podStartSLOduration=1.891737745 podStartE2EDuration="2.206084548s" totalImagesPullingTime="314.346803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:26.203398082 +0000 UTC m=+2048.228174213" watchObservedRunningTime="2026-10-29 22:48:26.206084548 +0000 UTC m=+2048.230860733"
Oct 29 22:48:26.650036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:26.650001    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:26.881461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:26.881425    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:26.881927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:26.881909    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:26.882551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:26.882530    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:48:28.346592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:28.346561    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:29.335961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:29.335883    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:29.530454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:29.530414    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx" podUID="70d957f1-c5a8-4993-a3af-2e35b2934cc4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:48:29.530978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:29.530882    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t" podUID="1e35c643-1248-47c8-8acd-1a4991f0a395" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:48:29.827632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:29.827448    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:29.851245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:29.851213    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"]
Oct 29 22:48:29.870109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:29.868398    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:29.925346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:29.914732    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:48:29.925346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:29.923616    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:48:29.961515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:29.944453    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqnt7\" (UniqueName: \"kubernetes.io/projected/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-kube-api-access-cqnt7\") pod \"maas-api-revoke-sub-a01097d0-66jxx\" (UID: \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:29.961515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:29.944569    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a01097d0-66jxx\" (UID: \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:29.961515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:29.944623    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a01097d0-66jxx\" (UID: \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:30.012765 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.003626    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"]
Oct 29 22:48:30.055347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.049616    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqnt7\" (UniqueName: \"kubernetes.io/projected/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-kube-api-access-cqnt7\") pod \"maas-api-revoke-sub-a01097d0-66jxx\" (UID: \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:30.055347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.050099    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a01097d0-66jxx\" (UID: \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:30.055347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.050151    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a01097d0-66jxx\" (UID: \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:30.055347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.051176    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a01097d0-66jxx\" (UID: \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:30.064030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.063992    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a01097d0-66jxx\" (UID: \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:30.078170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.078103    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqnt7\" (UniqueName: \"kubernetes.io/projected/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-kube-api-access-cqnt7\") pod \"maas-api-revoke-sub-a01097d0-66jxx\" (UID: \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:30.114981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.114885    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:30.329545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.329470    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:30.356723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.356690    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"]
Oct 29 22:48:30.364020 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.363999    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:30.377374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.377346    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"]
Oct 29 22:48:30.468445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.468410    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24tkl\" (UniqueName: \"kubernetes.io/projected/a0d4495a-f7e6-49ea-9a23-904a01a926e6-kube-api-access-24tkl\") pod \"maas-api-revoke-sub-ba5f8d01-kkgnp\" (UID: \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:30.468639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.468615    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba5f8d01-kkgnp\" (UID: \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:30.468714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.468683    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba5f8d01-kkgnp\" (UID: \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:30.570835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.570800    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba5f8d01-kkgnp\" (UID: \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:30.571255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.570915    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba5f8d01-kkgnp\" (UID: \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:30.571255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.571037    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24tkl\" (UniqueName: \"kubernetes.io/projected/a0d4495a-f7e6-49ea-9a23-904a01a926e6-kube-api-access-24tkl\") pod \"maas-api-revoke-sub-ba5f8d01-kkgnp\" (UID: \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:30.571755 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.571718    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba5f8d01-kkgnp\" (UID: \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:30.573424 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.573405    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba5f8d01-kkgnp\" (UID: \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:30.592012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.591963    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24tkl\" (UniqueName: \"kubernetes.io/projected/a0d4495a-f7e6-49ea-9a23-904a01a926e6-kube-api-access-24tkl\") pod \"maas-api-revoke-sub-ba5f8d01-kkgnp\" (UID: \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:30.647264 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:48:30.647236    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c6d23b_e3ce_4bf3_91e2_ecd2fd291a10.slice/crio-718f79430957bc9795b1308d0208f6355e9b7ea8efca11e4cd6d419c47401f74 WatchSource:0}: Error finding container 718f79430957bc9795b1308d0208f6355e9b7ea8efca11e4cd6d419c47401f74: Status 404 returned error can't find the container with id 718f79430957bc9795b1308d0208f6355e9b7ea8efca11e4cd6d419c47401f74
Oct 29 22:48:30.651717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.651692    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"]
Oct 29 22:48:30.694494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:30.694474    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:31.009951 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:48:31.009918    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d4495a_f7e6_49ea_9a23_904a01a926e6.slice/crio-72d65d64a26351bdfedc4fd57cc9bed88bc248cd703ec4559e06917fefde8fd5 WatchSource:0}: Error finding container 72d65d64a26351bdfedc4fd57cc9bed88bc248cd703ec4559e06917fefde8fd5: Status 404 returned error can't find the container with id 72d65d64a26351bdfedc4fd57cc9bed88bc248cd703ec4559e06917fefde8fd5
Oct 29 22:48:31.014910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:31.014880    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"]
Oct 29 22:48:31.230189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:31.230157    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp" event={"ID":"a0d4495a-f7e6-49ea-9a23-904a01a926e6","Type":"ContainerStarted","Data":"72d65d64a26351bdfedc4fd57cc9bed88bc248cd703ec4559e06917fefde8fd5"}
Oct 29 22:48:31.231909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:31.231886    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx" event={"ID":"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10","Type":"ContainerStarted","Data":"a4f74136f0154a6044c71d945abe071f89eff9853921656959a0d3b3e5fc49de"}
Oct 29 22:48:31.232033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:31.231932    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx" event={"ID":"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10","Type":"ContainerStarted","Data":"718f79430957bc9795b1308d0208f6355e9b7ea8efca11e4cd6d419c47401f74"}
Oct 29 22:48:31.276048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:31.276008    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx" podStartSLOduration=2.275995372 podStartE2EDuration="2.275995372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:31.272404794 +0000 UTC m=+2053.297180945" watchObservedRunningTime="2026-10-29 22:48:31.275995372 +0000 UTC m=+2053.300771503"
Oct 29 22:48:31.868479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:31.868441    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:32.174762 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:32.174737    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 22:48:32.242904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:32.242809    4860 generic.go:356] "Generic (PLEG): container finished" podID="a0d4495a-f7e6-49ea-9a23-904a01a926e6" containerID="ed64529fc35530baccaab812c57f5b581a85fe548892b848be8daea2e7fd6602" exitCode=0
Oct 29 22:48:32.243645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:32.242946    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp" event={"ID":"a0d4495a-f7e6-49ea-9a23-904a01a926e6","Type":"ContainerDied","Data":"ed64529fc35530baccaab812c57f5b581a85fe548892b848be8daea2e7fd6602"}
Oct 29 22:48:32.245183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:32.245152    4860 generic.go:356] "Generic (PLEG): container finished" podID="c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10" containerID="a4f74136f0154a6044c71d945abe071f89eff9853921656959a0d3b3e5fc49de" exitCode=0
Oct 29 22:48:32.245297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:32.245235    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx" event={"ID":"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10","Type":"ContainerDied","Data":"a4f74136f0154a6044c71d945abe071f89eff9853921656959a0d3b3e5fc49de"}
Oct 29 22:48:33.469469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.469426    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:33.554704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.553637    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:33.628380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.625268    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:33.640841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.640806    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-xks-ca\") pod \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\" (UID: \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\") "
Oct 29 22:48:33.641131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.641106    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-kube-api-access-cqnt7\" (UniqueName: \"kubernetes.io/projected/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-kube-api-access-cqnt7\") pod \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\" (UID: \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\") "
Oct 29 22:48:33.641232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.641213    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-openshift-ca\") pod \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\" (UID: \"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10\") "
Oct 29 22:48:33.642119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.642087    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-openshift-ca" pod "c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10" (UID: "c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:33.644568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.644540    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-xks-ca" pod "c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10" (UID: "c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:33.645177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.645148    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-kube-api-access-cqnt7" pod "c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10" (UID: "c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10"). InnerVolumeSpecName "kube-api-access-cqnt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:33.700352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.699455    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"]
Oct 29 22:48:33.700967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.700799    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a0d4495a-f7e6-49ea-9a23-904a01a926e6" containerName="revoke-keys"
Oct 29 22:48:33.700967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.700823    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d4495a-f7e6-49ea-9a23-904a01a926e6" containerName="revoke-keys"
Oct 29 22:48:33.700967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.700885    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10" containerName="revoke-keys"
Oct 29 22:48:33.700967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.700893    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10" containerName="revoke-keys"
Oct 29 22:48:33.701206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.701075    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d4495a-f7e6-49ea-9a23-904a01a926e6" containerName="revoke-keys"
Oct 29 22:48:33.701206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.701097    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10" containerName="revoke-keys"
Oct 29 22:48:33.710644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.710606    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:33.716048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.716019    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"]
Oct 29 22:48:33.743223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.743194    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d4495a-f7e6-49ea-9a23-904a01a926e6-kube-api-access-24tkl\" (UniqueName: \"kubernetes.io/projected/a0d4495a-f7e6-49ea-9a23-904a01a926e6-kube-api-access-24tkl\") pod \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\" (UID: \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\") "
Oct 29 22:48:33.743382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.743243    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-openshift-ca\") pod \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\" (UID: \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\") "
Oct 29 22:48:33.743471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.743383    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-xks-ca\") pod \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\" (UID: \"a0d4495a-f7e6-49ea-9a23-904a01a926e6\") "
Oct 29 22:48:33.744374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.743975    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-openshift-ca" pod "a0d4495a-f7e6-49ea-9a23-904a01a926e6" (UID: "a0d4495a-f7e6-49ea-9a23-904a01a926e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:33.744374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.744281    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:33.744374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.744303    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqnt7\" (UniqueName: \"kubernetes.io/projected/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-kube-api-access-cqnt7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:33.744374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.744317    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:33.744374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.744352    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:33.746473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.746442    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-xks-ca" pod "a0d4495a-f7e6-49ea-9a23-904a01a926e6" (UID: "a0d4495a-f7e6-49ea-9a23-904a01a926e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:33.747705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.747681    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d4495a-f7e6-49ea-9a23-904a01a926e6-kube-api-access-24tkl" pod "a0d4495a-f7e6-49ea-9a23-904a01a926e6" (UID: "a0d4495a-f7e6-49ea-9a23-904a01a926e6"). InnerVolumeSpecName "kube-api-access-24tkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:33.846034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.845950    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nwhm\" (UniqueName: \"kubernetes.io/projected/c0a25f99-c761-4c76-87c6-c8ace5698b17-kube-api-access-2nwhm\") pod \"maas-api-revoke-sub-aa82ad1b-dmkx6\" (UID: \"c0a25f99-c761-4c76-87c6-c8ace5698b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:33.846034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.846021    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa82ad1b-dmkx6\" (UID: \"c0a25f99-c761-4c76-87c6-c8ace5698b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:33.846408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.846379    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa82ad1b-dmkx6\" (UID: \"c0a25f99-c761-4c76-87c6-c8ace5698b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:33.846561 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.846546    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d4495a-f7e6-49ea-9a23-904a01a926e6-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:33.846629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.846565    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24tkl\" (UniqueName: \"kubernetes.io/projected/a0d4495a-f7e6-49ea-9a23-904a01a926e6-kube-api-access-24tkl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:33.948109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.948071    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa82ad1b-dmkx6\" (UID: \"c0a25f99-c761-4c76-87c6-c8ace5698b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:33.948279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.948188    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nwhm\" (UniqueName: \"kubernetes.io/projected/c0a25f99-c761-4c76-87c6-c8ace5698b17-kube-api-access-2nwhm\") pod \"maas-api-revoke-sub-aa82ad1b-dmkx6\" (UID: \"c0a25f99-c761-4c76-87c6-c8ace5698b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:33.948279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.948261    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa82ad1b-dmkx6\" (UID: \"c0a25f99-c761-4c76-87c6-c8ace5698b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:33.948833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.948806    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa82ad1b-dmkx6\" (UID: \"c0a25f99-c761-4c76-87c6-c8ace5698b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:33.950980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.950955    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa82ad1b-dmkx6\" (UID: \"c0a25f99-c761-4c76-87c6-c8ace5698b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:33.969212 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:33.969189    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nwhm\" (UniqueName: \"kubernetes.io/projected/c0a25f99-c761-4c76-87c6-c8ace5698b17-kube-api-access-2nwhm\") pod \"maas-api-revoke-sub-aa82ad1b-dmkx6\" (UID: \"c0a25f99-c761-4c76-87c6-c8ace5698b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:34.032557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:34.032531    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:34.268159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:34.268127    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"
Oct 29 22:48:34.268159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:34.268140    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp" event={"ID":"a0d4495a-f7e6-49ea-9a23-904a01a926e6","Type":"ContainerDied","Data":"72d65d64a26351bdfedc4fd57cc9bed88bc248cd703ec4559e06917fefde8fd5"}
Oct 29 22:48:34.268491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:34.268187    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72d65d64a26351bdfedc4fd57cc9bed88bc248cd703ec4559e06917fefde8fd5"
Oct 29 22:48:34.269850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:34.269822    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"
Oct 29 22:48:34.269984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:34.269858    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx" event={"ID":"c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10","Type":"ContainerDied","Data":"718f79430957bc9795b1308d0208f6355e9b7ea8efca11e4cd6d419c47401f74"}
Oct 29 22:48:34.269984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:34.269891    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="718f79430957bc9795b1308d0208f6355e9b7ea8efca11e4cd6d419c47401f74"
Oct 29 22:48:34.340685 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:48:34.340660    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a25f99_c761_4c76_87c6_c8ace5698b17.slice/crio-e86ad8b9266c229254357aaa2acc0b920f1898dcc8777269bc59ae55b6c9216e WatchSource:0}: Error finding container e86ad8b9266c229254357aaa2acc0b920f1898dcc8777269bc59ae55b6c9216e: Status 404 returned error can't find the container with id e86ad8b9266c229254357aaa2acc0b920f1898dcc8777269bc59ae55b6c9216e
Oct 29 22:48:34.346957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:34.346933    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"]
Oct 29 22:48:35.222123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:35.222094    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:35.314655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:35.314617    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6" event={"ID":"c0a25f99-c761-4c76-87c6-c8ace5698b17","Type":"ContainerStarted","Data":"4be32425b556c4a0ef733b12f03726db61e10bc531e5b70b3c62e604dd9ccd07"}
Oct 29 22:48:35.314860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:35.314706    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6" event={"ID":"c0a25f99-c761-4c76-87c6-c8ace5698b17","Type":"ContainerStarted","Data":"e86ad8b9266c229254357aaa2acc0b920f1898dcc8777269bc59ae55b6c9216e"}
Oct 29 22:48:35.361212 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:35.361158    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6" podStartSLOduration=2.36114484 podStartE2EDuration="2.36114484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:35.355460889 +0000 UTC m=+2057.380237045" watchObservedRunningTime="2026-10-29 22:48:35.36114484 +0000 UTC m=+2057.385920971"
Oct 29 22:48:36.326055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:36.326022    4860 generic.go:356] "Generic (PLEG): container finished" podID="c0a25f99-c761-4c76-87c6-c8ace5698b17" containerID="4be32425b556c4a0ef733b12f03726db61e10bc531e5b70b3c62e604dd9ccd07" exitCode=0
Oct 29 22:48:36.326471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:36.326068    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6" event={"ID":"c0a25f99-c761-4c76-87c6-c8ace5698b17","Type":"ContainerDied","Data":"4be32425b556c4a0ef733b12f03726db61e10bc531e5b70b3c62e604dd9ccd07"}
Oct 29 22:48:36.831420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:36.830991    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:36.923549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:36.923235    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"
Oct 29 22:48:36.955894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:36.955865    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"
Oct 29 22:48:37.016473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.016436    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e35c643-1248-47c8-8acd-1a4991f0a395-kube-api-access-9bksl\" (UniqueName: \"kubernetes.io/projected/1e35c643-1248-47c8-8acd-1a4991f0a395-kube-api-access-9bksl\") pod \"1e35c643-1248-47c8-8acd-1a4991f0a395\" (UID: \"1e35c643-1248-47c8-8acd-1a4991f0a395\") "
Oct 29 22:48:37.016657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.016567    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d957f1-c5a8-4993-a3af-2e35b2934cc4-kube-api-access-n6xtw\" (UniqueName: \"kubernetes.io/projected/70d957f1-c5a8-4993-a3af-2e35b2934cc4-kube-api-access-n6xtw\") pod \"70d957f1-c5a8-4993-a3af-2e35b2934cc4\" (UID: \"70d957f1-c5a8-4993-a3af-2e35b2934cc4\") "
Oct 29 22:48:37.016657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.016639    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70d957f1-c5a8-4993-a3af-2e35b2934cc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70d957f1-c5a8-4993-a3af-2e35b2934cc4-plugins-config-volume\") pod \"70d957f1-c5a8-4993-a3af-2e35b2934cc4\" (UID: \"70d957f1-c5a8-4993-a3af-2e35b2934cc4\") "
Oct 29 22:48:37.016940 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.016846    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e35c643-1248-47c8-8acd-1a4991f0a395-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e35c643-1248-47c8-8acd-1a4991f0a395-plugins-config-volume\") pod \"1e35c643-1248-47c8-8acd-1a4991f0a395\" (UID: \"1e35c643-1248-47c8-8acd-1a4991f0a395\") "
Oct 29 22:48:37.023868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.023824    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e35c643-1248-47c8-8acd-1a4991f0a395-plugins-config-volume" pod "1e35c643-1248-47c8-8acd-1a4991f0a395" (UID: "1e35c643-1248-47c8-8acd-1a4991f0a395"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:37.026778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.026746    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e35c643-1248-47c8-8acd-1a4991f0a395-kube-api-access-9bksl" pod "1e35c643-1248-47c8-8acd-1a4991f0a395" (UID: "1e35c643-1248-47c8-8acd-1a4991f0a395"). InnerVolumeSpecName "kube-api-access-9bksl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:37.029634 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.029603    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d957f1-c5a8-4993-a3af-2e35b2934cc4-kube-api-access-n6xtw" pod "70d957f1-c5a8-4993-a3af-2e35b2934cc4" (UID: "70d957f1-c5a8-4993-a3af-2e35b2934cc4"). InnerVolumeSpecName "kube-api-access-n6xtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:37.029975 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.029946    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d957f1-c5a8-4993-a3af-2e35b2934cc4-plugins-config-volume" pod "70d957f1-c5a8-4993-a3af-2e35b2934cc4" (UID: "70d957f1-c5a8-4993-a3af-2e35b2934cc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:37.124354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.124267    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e35c643-1248-47c8-8acd-1a4991f0a395-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:37.124354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.124295    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bksl\" (UniqueName: \"kubernetes.io/projected/1e35c643-1248-47c8-8acd-1a4991f0a395-kube-api-access-9bksl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:37.124354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.124304    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6xtw\" (UniqueName: \"kubernetes.io/projected/70d957f1-c5a8-4993-a3af-2e35b2934cc4-kube-api-access-n6xtw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:37.124354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.124313    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70d957f1-c5a8-4993-a3af-2e35b2934cc4-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:37.336290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.336258    4860 generic.go:356] "Generic (PLEG): container finished" podID="70d957f1-c5a8-4993-a3af-2e35b2934cc4" containerID="a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce" exitCode=0
Oct 29 22:48:37.336723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.336316    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"
Oct 29 22:48:37.336723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.336353    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx" event={"ID":"70d957f1-c5a8-4993-a3af-2e35b2934cc4","Type":"ContainerDied","Data":"a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce"}
Oct 29 22:48:37.336723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.336404    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx" event={"ID":"70d957f1-c5a8-4993-a3af-2e35b2934cc4","Type":"ContainerDied","Data":"6fbd78474f575ac65d790907241eea125503839bc95f6be33f22da7135716dc8"}
Oct 29 22:48:37.336723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.336432    4860 scope.go:122] "RemoveContainer" containerID="a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce"
Oct 29 22:48:37.338649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.338620    4860 generic.go:356] "Generic (PLEG): container finished" podID="1e35c643-1248-47c8-8acd-1a4991f0a395" containerID="1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652" exitCode=0
Oct 29 22:48:37.338767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.338682    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"
Oct 29 22:48:37.338767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.338692    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t" event={"ID":"1e35c643-1248-47c8-8acd-1a4991f0a395","Type":"ContainerDied","Data":"1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652"}
Oct 29 22:48:37.338767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.338750    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t" event={"ID":"1e35c643-1248-47c8-8acd-1a4991f0a395","Type":"ContainerDied","Data":"8ad703d20d0542b9385f64bbaeedbab4a72fd2c14f6985e5f145e406e0e30a8b"}
Oct 29 22:48:37.351106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.351087    4860 scope.go:122] "RemoveContainer" containerID="a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce"
Oct 29 22:48:37.351415 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:48:37.351395    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce\": container with ID starting with a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce not found: ID does not exist" containerID="a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce"
Oct 29 22:48:37.351493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.351421    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce"} err="failed to get container status \"a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce\": rpc error: code = NotFound desc = could not find container \"a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce\": container with ID starting with a0efd43d9efca6bc82e73a09d053b9bf0f6c19104a0a67cb117fcfbb447127ce not found: ID does not exist"
Oct 29 22:48:37.351493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.351437    4860 scope.go:122] "RemoveContainer" containerID="1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652"
Oct 29 22:48:37.363594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.363569    4860 scope.go:122] "RemoveContainer" containerID="1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652"
Oct 29 22:48:37.363926 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:48:37.363907    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652\": container with ID starting with 1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652 not found: ID does not exist" containerID="1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652"
Oct 29 22:48:37.363985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.363933    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652"} err="failed to get container status \"1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652\": rpc error: code = NotFound desc = could not find container \"1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652\": container with ID starting with 1fa78beb8f09934f6552540f627b47c8935ff4681977e76fcecdfe6fb8d42652 not found: ID does not exist"
Oct 29 22:48:37.396378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.396285    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"]
Oct 29 22:48:37.400955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.400926    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fee294-79797dc7c4-4q6tx"]
Oct 29 22:48:37.443076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.443045    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"]
Oct 29 22:48:37.448871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.448849    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fee294-c499455c5-ds72t"]
Oct 29 22:48:37.499162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.499138    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:37.529888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.529865    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a25f99-c761-4c76-87c6-c8ace5698b17-kube-api-access-2nwhm\" (UniqueName: \"kubernetes.io/projected/c0a25f99-c761-4c76-87c6-c8ace5698b17-kube-api-access-2nwhm\") pod \"c0a25f99-c761-4c76-87c6-c8ace5698b17\" (UID: \"c0a25f99-c761-4c76-87c6-c8ace5698b17\") "
Oct 29 22:48:37.530127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.530112    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-xks-ca\") pod \"c0a25f99-c761-4c76-87c6-c8ace5698b17\" (UID: \"c0a25f99-c761-4c76-87c6-c8ace5698b17\") "
Oct 29 22:48:37.530214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.530150    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-openshift-ca\") pod \"c0a25f99-c761-4c76-87c6-c8ace5698b17\" (UID: \"c0a25f99-c761-4c76-87c6-c8ace5698b17\") "
Oct 29 22:48:37.530584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.530557    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-openshift-ca" pod "c0a25f99-c761-4c76-87c6-c8ace5698b17" (UID: "c0a25f99-c761-4c76-87c6-c8ace5698b17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:37.530719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.530704    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:37.532019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.531995    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a25f99-c761-4c76-87c6-c8ace5698b17-kube-api-access-2nwhm" pod "c0a25f99-c761-4c76-87c6-c8ace5698b17" (UID: "c0a25f99-c761-4c76-87c6-c8ace5698b17"). InnerVolumeSpecName "kube-api-access-2nwhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:37.532194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.532176    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-xks-ca" pod "c0a25f99-c761-4c76-87c6-c8ace5698b17" (UID: "c0a25f99-c761-4c76-87c6-c8ace5698b17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:37.631856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.631828    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a25f99-c761-4c76-87c6-c8ace5698b17-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:37.631978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:37.631866    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nwhm\" (UniqueName: \"kubernetes.io/projected/c0a25f99-c761-4c76-87c6-c8ace5698b17-kube-api-access-2nwhm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:38.113918 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:38.113885    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e35c643-1248-47c8-8acd-1a4991f0a395" path="/var/lib/kubelet/pods/1e35c643-1248-47c8-8acd-1a4991f0a395/volumes"
Oct 29 22:48:38.114362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:38.114317    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d957f1-c5a8-4993-a3af-2e35b2934cc4" path="/var/lib/kubelet/pods/70d957f1-c5a8-4993-a3af-2e35b2934cc4/volumes"
Oct 29 22:48:38.351178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:38.351144    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6" event={"ID":"c0a25f99-c761-4c76-87c6-c8ace5698b17","Type":"ContainerDied","Data":"e86ad8b9266c229254357aaa2acc0b920f1898dcc8777269bc59ae55b6c9216e"}
Oct 29 22:48:38.351665 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:38.351228    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e86ad8b9266c229254357aaa2acc0b920f1898dcc8777269bc59ae55b6c9216e"
Oct 29 22:48:38.351665 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:38.351149    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"
Oct 29 22:48:38.356031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:38.356002    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:40.062039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:40.062004    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:40.204409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:40.204370    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:41.935453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:41.935416    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:43.428643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:43.428608    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:43.994588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:43.994554    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"]
Oct 29 22:48:43.995304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:43.995288    4860 cpu_manager.go:414] "containerMap: removing container" podUID="70d957f1-c5a8-4993-a3af-2e35b2934cc4" containerName="payload-processing"
Oct 29 22:48:43.995385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:43.995306    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d957f1-c5a8-4993-a3af-2e35b2934cc4" containerName="payload-processing"
Oct 29 22:48:43.995385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:43.995370    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1e35c643-1248-47c8-8acd-1a4991f0a395" containerName="payload-pre-processing"
Oct 29 22:48:43.995385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:43.995377    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e35c643-1248-47c8-8acd-1a4991f0a395" containerName="payload-pre-processing"
Oct 29 22:48:43.995491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:43.995402    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c0a25f99-c761-4c76-87c6-c8ace5698b17" containerName="revoke-keys"
Oct 29 22:48:43.995491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:43.995407    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a25f99-c761-4c76-87c6-c8ace5698b17" containerName="revoke-keys"
Oct 29 22:48:43.995556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:43.995540    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e35c643-1248-47c8-8acd-1a4991f0a395" containerName="payload-pre-processing"
Oct 29 22:48:43.995556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:43.995553    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a25f99-c761-4c76-87c6-c8ace5698b17" containerName="revoke-keys"
Oct 29 22:48:43.995625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:43.995561    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d957f1-c5a8-4993-a3af-2e35b2934cc4" containerName="payload-processing"
Oct 29 22:48:44.002059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.002029    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:44.017803 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.017767    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"]
Oct 29 22:48:44.024843 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.024821    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:48:44.025074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.025049    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:48:44.125589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.125559    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e6d451d-8frgz\" (UID: \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:44.125768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.125739    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k66mm\" (UniqueName: \"kubernetes.io/projected/f67418cf-95b9-44fd-bbef-aa3052ee6e51-kube-api-access-k66mm\") pod \"maas-api-revoke-sub-2e6d451d-8frgz\" (UID: \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:44.125861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.125841    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e6d451d-8frgz\" (UID: \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:44.227398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.227363    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k66mm\" (UniqueName: \"kubernetes.io/projected/f67418cf-95b9-44fd-bbef-aa3052ee6e51-kube-api-access-k66mm\") pod \"maas-api-revoke-sub-2e6d451d-8frgz\" (UID: \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:44.227568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.227433    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e6d451d-8frgz\" (UID: \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:44.227645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.227627    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e6d451d-8frgz\" (UID: \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:44.228253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.228229    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e6d451d-8frgz\" (UID: \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:44.229651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.229630    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e6d451d-8frgz\" (UID: \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:44.248201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.248145    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k66mm\" (UniqueName: \"kubernetes.io/projected/f67418cf-95b9-44fd-bbef-aa3052ee6e51-kube-api-access-k66mm\") pod \"maas-api-revoke-sub-2e6d451d-8frgz\" (UID: \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:44.324853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.324831    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:44.626711 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:48:44.626680    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67418cf_95b9_44fd_bbef_aa3052ee6e51.slice/crio-e125167ed08972c6665c6f6c9f62f8cce92d4f2964dd6acf2ee1c259cd321642 WatchSource:0}: Error finding container e125167ed08972c6665c6f6c9f62f8cce92d4f2964dd6acf2ee1c259cd321642: Status 404 returned error can't find the container with id e125167ed08972c6665c6f6c9f62f8cce92d4f2964dd6acf2ee1c259cd321642
Oct 29 22:48:44.631037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:44.631012    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"]
Oct 29 22:48:45.146662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:45.146590    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:45.427052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:45.426958    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz" event={"ID":"f67418cf-95b9-44fd-bbef-aa3052ee6e51","Type":"ContainerStarted","Data":"f8d10ecbbd23019d2a287e4f2b31299805a6342d495c6a6b1eb04842ce93d2f0"}
Oct 29 22:48:45.427052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:45.427020    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz" event={"ID":"f67418cf-95b9-44fd-bbef-aa3052ee6e51","Type":"ContainerStarted","Data":"e125167ed08972c6665c6f6c9f62f8cce92d4f2964dd6acf2ee1c259cd321642"}
Oct 29 22:48:45.472991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:45.472940    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz" podStartSLOduration=2.472925341 podStartE2EDuration="2.472925341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:45.470463998 +0000 UTC m=+2067.495240130" watchObservedRunningTime="2026-10-29 22:48:45.472925341 +0000 UTC m=+2067.497701478"
Oct 29 22:48:45.967770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:45.967736    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:46.095281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.095242    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"]
Oct 29 22:48:46.108407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.101595    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:46.155873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.155835    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvsg\" (UniqueName: \"kubernetes.io/projected/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-kube-api-access-zlvsg\") pod \"maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r\" (UID: \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:46.156106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.155888    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r\" (UID: \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:46.156106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.156046    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r\" (UID: \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:46.162905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.162878    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"]
Oct 29 22:48:46.258120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.258041    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvsg\" (UniqueName: \"kubernetes.io/projected/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-kube-api-access-zlvsg\") pod \"maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r\" (UID: \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:46.258120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.258086    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r\" (UID: \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:46.258391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.258239    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r\" (UID: \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:46.260795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.259465    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r\" (UID: \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:46.261651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.261629    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r\" (UID: \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:46.286993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.286965    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvsg\" (UniqueName: \"kubernetes.io/projected/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-kube-api-access-zlvsg\") pod \"maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r\" (UID: \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:46.342461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.342433    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:46.453545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.453508    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:46.464810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.464771    4860 generic.go:356] "Generic (PLEG): container finished" podID="f67418cf-95b9-44fd-bbef-aa3052ee6e51" containerID="f8d10ecbbd23019d2a287e4f2b31299805a6342d495c6a6b1eb04842ce93d2f0" exitCode=0
Oct 29 22:48:46.464980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.464864    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz" event={"ID":"f67418cf-95b9-44fd-bbef-aa3052ee6e51","Type":"ContainerDied","Data":"f8d10ecbbd23019d2a287e4f2b31299805a6342d495c6a6b1eb04842ce93d2f0"}
Oct 29 22:48:46.509787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.509695    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"]
Oct 29 22:48:46.516289 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.516264    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:46.536885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.536376    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"]
Oct 29 22:48:46.666614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.666574    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p99tn\" (UniqueName: \"kubernetes.io/projected/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-kube-api-access-p99tn\") pod \"maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z\" (UID: \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:46.666767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.666726    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z\" (UID: \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:46.666851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.666835    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z\" (UID: \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:46.768009 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.767929    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p99tn\" (UniqueName: \"kubernetes.io/projected/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-kube-api-access-p99tn\") pod \"maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z\" (UID: \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:46.768156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.768034    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z\" (UID: \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:46.768156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.768105    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z\" (UID: \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:46.768708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.768688    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z\" (UID: \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:46.770707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.770680    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z\" (UID: \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:46.788452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.788425    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p99tn\" (UniqueName: \"kubernetes.io/projected/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-kube-api-access-p99tn\") pod \"maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z\" (UID: \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:46.798803 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:48:46.798767    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846f68a1_8fc5_4a3d_b1b6_b65ce70b4fa9.slice/crio-0a2ffcfe1e9825d8bdb81a53da7b1fff26ca99e85ac02c867d638348004c90c1 WatchSource:0}: Error finding container 0a2ffcfe1e9825d8bdb81a53da7b1fff26ca99e85ac02c867d638348004c90c1: Status 404 returned error can't find the container with id 0a2ffcfe1e9825d8bdb81a53da7b1fff26ca99e85ac02c867d638348004c90c1
Oct 29 22:48:46.803053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.803032    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"]
Oct 29 22:48:46.844526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.844494    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:46.997153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:46.997118    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:47.146218 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:48:47.146186    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f07d524_ac1c_47cd_ab07_105ebc7eec0b.slice/crio-6a70e58c20a99306d0e705e6b2d66fba618742612da46241990dda6d34b323f2 WatchSource:0}: Error finding container 6a70e58c20a99306d0e705e6b2d66fba618742612da46241990dda6d34b323f2: Status 404 returned error can't find the container with id 6a70e58c20a99306d0e705e6b2d66fba618742612da46241990dda6d34b323f2
Oct 29 22:48:47.150316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.150292    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"]
Oct 29 22:48:47.475951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.475917    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z" event={"ID":"7f07d524-ac1c-47cd-ab07-105ebc7eec0b","Type":"ContainerStarted","Data":"6a70e58c20a99306d0e705e6b2d66fba618742612da46241990dda6d34b323f2"}
Oct 29 22:48:47.477734 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.477703    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r" event={"ID":"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9","Type":"ContainerStarted","Data":"70262766aa27b24d779b05ef9c427579f7dc0a1e0248735bcd81c8b9c0078ff0"}
Oct 29 22:48:47.477874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.477760    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r" event={"ID":"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9","Type":"ContainerStarted","Data":"0a2ffcfe1e9825d8bdb81a53da7b1fff26ca99e85ac02c867d638348004c90c1"}
Oct 29 22:48:47.523170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.523111    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r" podStartSLOduration=1.5230968040000001 podStartE2EDuration="1.523096804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:47.519410511 +0000 UTC m=+2069.544186665" watchObservedRunningTime="2026-10-29 22:48:47.523096804 +0000 UTC m=+2069.547872934"
Oct 29 22:48:47.662751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.662723    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:47.785445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.785416    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-xks-ca\") pod \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\" (UID: \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\") "
Oct 29 22:48:47.785636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.785621    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67418cf-95b9-44fd-bbef-aa3052ee6e51-kube-api-access-k66mm\" (UniqueName: \"kubernetes.io/projected/f67418cf-95b9-44fd-bbef-aa3052ee6e51-kube-api-access-k66mm\") pod \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\" (UID: \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\") "
Oct 29 22:48:47.785712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.785698    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-openshift-ca\") pod \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\" (UID: \"f67418cf-95b9-44fd-bbef-aa3052ee6e51\") "
Oct 29 22:48:47.786118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.786090    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-openshift-ca" pod "f67418cf-95b9-44fd-bbef-aa3052ee6e51" (UID: "f67418cf-95b9-44fd-bbef-aa3052ee6e51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:47.786513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.786383    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:47.787649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.787627    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-xks-ca" pod "f67418cf-95b9-44fd-bbef-aa3052ee6e51" (UID: "f67418cf-95b9-44fd-bbef-aa3052ee6e51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:47.787799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.787778    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67418cf-95b9-44fd-bbef-aa3052ee6e51-kube-api-access-k66mm" pod "f67418cf-95b9-44fd-bbef-aa3052ee6e51" (UID: "f67418cf-95b9-44fd-bbef-aa3052ee6e51"). InnerVolumeSpecName "kube-api-access-k66mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:47.887457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.887430    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k66mm\" (UniqueName: \"kubernetes.io/projected/f67418cf-95b9-44fd-bbef-aa3052ee6e51-kube-api-access-k66mm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:47.887457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:47.887455    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67418cf-95b9-44fd-bbef-aa3052ee6e51-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:48.059888 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:48:48.059792    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:48:48.488990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:48.488959    4860 generic.go:356] "Generic (PLEG): container finished" podID="846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9" containerID="70262766aa27b24d779b05ef9c427579f7dc0a1e0248735bcd81c8b9c0078ff0" exitCode=0
Oct 29 22:48:48.489155 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:48.489043    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r" event={"ID":"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9","Type":"ContainerDied","Data":"70262766aa27b24d779b05ef9c427579f7dc0a1e0248735bcd81c8b9c0078ff0"}
Oct 29 22:48:48.490915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:48.490892    4860 generic.go:356] "Generic (PLEG): container finished" podID="7f07d524-ac1c-47cd-ab07-105ebc7eec0b" containerID="c970869790f420de57166e1cda1055a04123537c10fb0cf31b64d9eaad59929f" exitCode=0
Oct 29 22:48:48.491036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:48.490971    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z" event={"ID":"7f07d524-ac1c-47cd-ab07-105ebc7eec0b","Type":"ContainerDied","Data":"c970869790f420de57166e1cda1055a04123537c10fb0cf31b64d9eaad59929f"}
Oct 29 22:48:48.497172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:48.497150    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"
Oct 29 22:48:48.497303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:48.497181    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz" event={"ID":"f67418cf-95b9-44fd-bbef-aa3052ee6e51","Type":"ContainerDied","Data":"e125167ed08972c6665c6f6c9f62f8cce92d4f2964dd6acf2ee1c259cd321642"}
Oct 29 22:48:48.497303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:48.497224    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e125167ed08972c6665c6f6c9f62f8cce92d4f2964dd6acf2ee1c259cd321642"
Oct 29 22:48:48.649413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:48.649375    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:49.721066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.721039    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:49.725426 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.725407    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:49.820902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.820806    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-kube-api-access-zlvsg\" (UniqueName: \"kubernetes.io/projected/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-kube-api-access-zlvsg\") pod \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\" (UID: \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\") "
Oct 29 22:48:49.821088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.820955    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-xks-ca\") pod \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\" (UID: \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\") "
Oct 29 22:48:49.821088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.820993    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-xks-ca\") pod \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\" (UID: \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\") "
Oct 29 22:48:49.821088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.821050    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-openshift-ca\") pod \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\" (UID: \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\") "
Oct 29 22:48:49.821276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.821095    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-kube-api-access-p99tn\" (UniqueName: \"kubernetes.io/projected/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-kube-api-access-p99tn\") pod \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\" (UID: \"7f07d524-ac1c-47cd-ab07-105ebc7eec0b\") "
Oct 29 22:48:49.821391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.821293    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-openshift-ca\") pod \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\" (UID: \"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9\") "
Oct 29 22:48:49.822474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.822439    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-openshift-ca" pod "7f07d524-ac1c-47cd-ab07-105ebc7eec0b" (UID: "7f07d524-ac1c-47cd-ab07-105ebc7eec0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:49.823587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.822794    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-openshift-ca" pod "846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9" (UID: "846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:48:49.824620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.824592    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-xks-ca" pod "846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9" (UID: "846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:49.824799 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.824769    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-kube-api-access-zlvsg" pod "846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9" (UID: "846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9"). InnerVolumeSpecName "kube-api-access-zlvsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:49.824917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.824849    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-kube-api-access-p99tn" pod "7f07d524-ac1c-47cd-ab07-105ebc7eec0b" (UID: "7f07d524-ac1c-47cd-ab07-105ebc7eec0b"). InnerVolumeSpecName "kube-api-access-p99tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:48:49.825952 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.825933    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-xks-ca" pod "7f07d524-ac1c-47cd-ab07-105ebc7eec0b" (UID: "7f07d524-ac1c-47cd-ab07-105ebc7eec0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:48:49.924791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.924579    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlvsg\" (UniqueName: \"kubernetes.io/projected/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-kube-api-access-zlvsg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:49.924791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.924617    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:49.924791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.924631    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:49.924791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.924645    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:49.924791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.924658    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p99tn\" (UniqueName: \"kubernetes.io/projected/7f07d524-ac1c-47cd-ab07-105ebc7eec0b-kube-api-access-p99tn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:49.924791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.924670    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:48:49.982363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:49.982334    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:50.529998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:50.527403    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"
Oct 29 22:48:50.529998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:50.527435    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z" event={"ID":"7f07d524-ac1c-47cd-ab07-105ebc7eec0b","Type":"ContainerDied","Data":"6a70e58c20a99306d0e705e6b2d66fba618742612da46241990dda6d34b323f2"}
Oct 29 22:48:50.529998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:50.527498    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a70e58c20a99306d0e705e6b2d66fba618742612da46241990dda6d34b323f2"
Oct 29 22:48:50.531879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:50.531852    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"
Oct 29 22:48:50.532035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:50.531853    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r" event={"ID":"846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9","Type":"ContainerDied","Data":"0a2ffcfe1e9825d8bdb81a53da7b1fff26ca99e85ac02c867d638348004c90c1"}
Oct 29 22:48:50.532035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:50.531968    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a2ffcfe1e9825d8bdb81a53da7b1fff26ca99e85ac02c867d638348004c90c1"
Oct 29 22:48:50.643315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:50.643279    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:50.778492 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:50.778462    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"]
Oct 29 22:48:50.783768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:50.783656    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:50.785114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:50.785088    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fee294-q6bjc"]
Oct 29 22:48:50.915243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:50.915207    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:51.273551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:51.273522    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:52.114757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:52.114724    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a7b7f8-7f10-445f-8e90-af03ec8f8498" path="/var/lib/kubelet/pods/40a7b7f8-7f10-445f-8e90-af03ec8f8498/volumes"
Oct 29 22:48:52.637997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:52.637962    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:54.478847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:54.478809    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:56.389042 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.389002    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:56.406791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.406749    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"]
Oct 29 22:48:56.436436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.425580    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f67418cf-95b9-44fd-bbef-aa3052ee6e51" containerName="revoke-keys"
Oct 29 22:48:56.436684 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.436657    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67418cf-95b9-44fd-bbef-aa3052ee6e51" containerName="revoke-keys"
Oct 29 22:48:56.436857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.436845    4860 cpu_manager.go:414] "containerMap: removing container" podUID="7f07d524-ac1c-47cd-ab07-105ebc7eec0b" containerName="revoke-keys"
Oct 29 22:48:56.436948 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.436937    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f07d524-ac1c-47cd-ab07-105ebc7eec0b" containerName="revoke-keys"
Oct 29 22:48:56.437055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.437046    4860 cpu_manager.go:414] "containerMap: removing container" podUID="846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9" containerName="revoke-keys"
Oct 29 22:48:56.437123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.437115    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9" containerName="revoke-keys"
Oct 29 22:48:56.437469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.437450    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9" containerName="revoke-keys"
Oct 29 22:48:56.437590 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.437581    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67418cf-95b9-44fd-bbef-aa3052ee6e51" containerName="revoke-keys"
Oct 29 22:48:56.437686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.437675    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f07d524-ac1c-47cd-ab07-105ebc7eec0b" containerName="revoke-keys"
Oct 29 22:48:56.447695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.447557    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"]
Oct 29 22:48:56.447965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.447892    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:48:56.467344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.467302    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:48:56.467523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.467366    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:48:56.519222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.519183    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:56.605342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.604700    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-268zc\" (UniqueName: \"kubernetes.io/projected/46fd8747-b6ca-4dd6-a91e-e03851420f04-kube-api-access-268zc\") pod \"maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4\" (UID: \"46fd8747-b6ca-4dd6-a91e-e03851420f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:48:56.605342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.604895    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4\" (UID: \"46fd8747-b6ca-4dd6-a91e-e03851420f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:48:56.605342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.605006    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4\" (UID: \"46fd8747-b6ca-4dd6-a91e-e03851420f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:48:56.721366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.716657    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-268zc\" (UniqueName: \"kubernetes.io/projected/46fd8747-b6ca-4dd6-a91e-e03851420f04-kube-api-access-268zc\") pod \"maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4\" (UID: \"46fd8747-b6ca-4dd6-a91e-e03851420f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:48:56.721366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.716843    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4\" (UID: \"46fd8747-b6ca-4dd6-a91e-e03851420f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:48:56.721366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.716926    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4\" (UID: \"46fd8747-b6ca-4dd6-a91e-e03851420f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:48:56.721366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.717973    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4\" (UID: \"46fd8747-b6ca-4dd6-a91e-e03851420f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:48:56.736363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.736308    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4\" (UID: \"46fd8747-b6ca-4dd6-a91e-e03851420f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:48:56.739780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.739754    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-268zc\" (UniqueName: \"kubernetes.io/projected/46fd8747-b6ca-4dd6-a91e-e03851420f04-kube-api-access-268zc\") pod \"maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4\" (UID: \"46fd8747-b6ca-4dd6-a91e-e03851420f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:48:56.785800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:56.785763    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:48:57.559421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:57.559389    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"]
Oct 29 22:48:57.718666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:57.718620    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:57.719549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:57.719517    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4" event={"ID":"46fd8747-b6ca-4dd6-a91e-e03851420f04","Type":"ContainerStarted","Data":"048c2142caa44941e9113b3e5e0597b11b41a9cb93189f9c8ea70448770f1fc5"}
Oct 29 22:48:57.770348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:57.768825    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"]
Oct 29 22:48:57.781346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:57.779547    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"]
Oct 29 22:48:57.855455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:57.854126    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:58.024513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.024377    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:58.036661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.036623    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"]
Oct 29 22:48:58.036949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.036916    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy" containerID="cri-o://9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4" gracePeriod=30
Oct 29 22:48:58.155663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.155577    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:58.541640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.541579    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:58.603525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.603432    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"]
Oct 29 22:48:58.622833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.622805    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:48:58.643788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.643757    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"]
Oct 29 22:48:58.663380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.661802    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zm5\" (UniqueName: \"kubernetes.io/projected/13104ade-e48c-489a-8d08-254b7f211239-kube-api-access-f9zm5\") pod \"maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5\" (UID: \"13104ade-e48c-489a-8d08-254b7f211239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:48:58.663380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.661883    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13104ade-e48c-489a-8d08-254b7f211239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5\" (UID: \"13104ade-e48c-489a-8d08-254b7f211239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:48:58.663380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.661921    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13104ade-e48c-489a-8d08-254b7f211239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5\" (UID: \"13104ade-e48c-489a-8d08-254b7f211239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:48:58.731266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.731225    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4" event={"ID":"46fd8747-b6ca-4dd6-a91e-e03851420f04","Type":"ContainerStarted","Data":"c31b728797345e08402d9b25b8926818d530d8328878e689b413e569e3b6d854"}
Oct 29 22:48:58.762867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.762832    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zm5\" (UniqueName: \"kubernetes.io/projected/13104ade-e48c-489a-8d08-254b7f211239-kube-api-access-f9zm5\") pod \"maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5\" (UID: \"13104ade-e48c-489a-8d08-254b7f211239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:48:58.763076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.762911    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13104ade-e48c-489a-8d08-254b7f211239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5\" (UID: \"13104ade-e48c-489a-8d08-254b7f211239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:48:58.763076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.762954    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13104ade-e48c-489a-8d08-254b7f211239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5\" (UID: \"13104ade-e48c-489a-8d08-254b7f211239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:48:58.763740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.763719    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13104ade-e48c-489a-8d08-254b7f211239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5\" (UID: \"13104ade-e48c-489a-8d08-254b7f211239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:48:58.765398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.765370    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13104ade-e48c-489a-8d08-254b7f211239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5\" (UID: \"13104ade-e48c-489a-8d08-254b7f211239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:48:58.777240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.777183    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4" podStartSLOduration=2.777166315 podStartE2EDuration="2.777166315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:58.773389163 +0000 UTC m=+2080.798165314" watchObservedRunningTime="2026-10-29 22:48:58.777166315 +0000 UTC m=+2080.801942446"
Oct 29 22:48:58.786455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.786427    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zm5\" (UniqueName: \"kubernetes.io/projected/13104ade-e48c-489a-8d08-254b7f211239-kube-api-access-f9zm5\") pod \"maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5\" (UID: \"13104ade-e48c-489a-8d08-254b7f211239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:48:58.949593 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:58.949558    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:48:59.125683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:59.125611    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"]
Oct 29 22:48:59.261278 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:48:59.261250    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13104ade_e48c_489a_8d08_254b7f211239.slice/crio-3c9b19aaaa288cb7ca022c39e87c01bfe2d4acb9d58b293dbdd5059f8c00791b WatchSource:0}: Error finding container 3c9b19aaaa288cb7ca022c39e87c01bfe2d4acb9d58b293dbdd5059f8c00791b: Status 404 returned error can't find the container with id 3c9b19aaaa288cb7ca022c39e87c01bfe2d4acb9d58b293dbdd5059f8c00791b
Oct 29 22:48:59.264944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:59.264922    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"]
Oct 29 22:48:59.268088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:59.268044    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:59.393542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:59.393441    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:59.842350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:59.835520    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5" event={"ID":"13104ade-e48c-489a-8d08-254b7f211239","Type":"ContainerStarted","Data":"248f0ac7830832180fd0704ef77230bdc7a3c64fa6c6e3195d5583ff6874c4db"}
Oct 29 22:48:59.842350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:59.835608    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5" event={"ID":"13104ade-e48c-489a-8d08-254b7f211239","Type":"ContainerStarted","Data":"3c9b19aaaa288cb7ca022c39e87c01bfe2d4acb9d58b293dbdd5059f8c00791b"}
Oct 29 22:48:59.864373 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:59.853313    4860 generic.go:356] "Generic (PLEG): container finished" podID="46fd8747-b6ca-4dd6-a91e-e03851420f04" containerID="c31b728797345e08402d9b25b8926818d530d8328878e689b413e569e3b6d854" exitCode=0
Oct 29 22:48:59.864373 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:59.853577    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4" event={"ID":"46fd8747-b6ca-4dd6-a91e-e03851420f04","Type":"ContainerDied","Data":"c31b728797345e08402d9b25b8926818d530d8328878e689b413e569e3b6d854"}
Oct 29 22:48:59.939252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:48:59.939191    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5" podStartSLOduration=1.93917681 podStartE2EDuration="1.93917681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:59.895243975 +0000 UTC m=+2081.920020120" watchObservedRunningTime="2026-10-29 22:48:59.93917681 +0000 UTC m=+2081.963952951"
Oct 29 22:49:00.249273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:00.249195    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:00.865638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:00.865604    4860 generic.go:356] "Generic (PLEG): container finished" podID="13104ade-e48c-489a-8d08-254b7f211239" containerID="248f0ac7830832180fd0704ef77230bdc7a3c64fa6c6e3195d5583ff6874c4db" exitCode=0
Oct 29 22:49:00.866102 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:00.865644    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5" event={"ID":"13104ade-e48c-489a-8d08-254b7f211239","Type":"ContainerDied","Data":"248f0ac7830832180fd0704ef77230bdc7a3c64fa6c6e3195d5583ff6874c4db"}
Oct 29 22:49:01.052570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.052542    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:49:01.115232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.115203    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-xks-ca\") pod \"46fd8747-b6ca-4dd6-a91e-e03851420f04\" (UID: \"46fd8747-b6ca-4dd6-a91e-e03851420f04\") "
Oct 29 22:49:01.115431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.115361    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-openshift-ca\") pod \"46fd8747-b6ca-4dd6-a91e-e03851420f04\" (UID: \"46fd8747-b6ca-4dd6-a91e-e03851420f04\") "
Oct 29 22:49:01.115532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.115469    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fd8747-b6ca-4dd6-a91e-e03851420f04-kube-api-access-268zc\" (UniqueName: \"kubernetes.io/projected/46fd8747-b6ca-4dd6-a91e-e03851420f04-kube-api-access-268zc\") pod \"46fd8747-b6ca-4dd6-a91e-e03851420f04\" (UID: \"46fd8747-b6ca-4dd6-a91e-e03851420f04\") "
Oct 29 22:49:01.115784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.115727    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-openshift-ca" pod "46fd8747-b6ca-4dd6-a91e-e03851420f04" (UID: "46fd8747-b6ca-4dd6-a91e-e03851420f04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:01.116834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.116172    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.117567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.117540    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-xks-ca" pod "46fd8747-b6ca-4dd6-a91e-e03851420f04" (UID: "46fd8747-b6ca-4dd6-a91e-e03851420f04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:01.117761 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.117739    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fd8747-b6ca-4dd6-a91e-e03851420f04-kube-api-access-268zc" pod "46fd8747-b6ca-4dd6-a91e-e03851420f04" (UID: "46fd8747-b6ca-4dd6-a91e-e03851420f04"). InnerVolumeSpecName "kube-api-access-268zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:01.217379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.217347    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-268zc\" (UniqueName: \"kubernetes.io/projected/46fd8747-b6ca-4dd6-a91e-e03851420f04-kube-api-access-268zc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.217379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.217380    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46fd8747-b6ca-4dd6-a91e-e03851420f04-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:01.875916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.875888    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"
Oct 29 22:49:01.876439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.875931    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4" event={"ID":"46fd8747-b6ca-4dd6-a91e-e03851420f04","Type":"ContainerDied","Data":"048c2142caa44941e9113b3e5e0597b11b41a9cb93189f9c8ea70448770f1fc5"}
Oct 29 22:49:01.876439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.875979    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="048c2142caa44941e9113b3e5e0597b11b41a9cb93189f9c8ea70448770f1fc5"
Oct 29 22:49:01.997597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:01.997560    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:02.052611 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.051567    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"]
Oct 29 22:49:02.059853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.059820    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3234dde-rvgp9"]
Oct 29 22:49:02.070273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.070246    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:49:02.115492 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.115442    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2" path="/var/lib/kubelet/pods/cd7ba7cd-6f95-44bc-b8d7-5bb61e6d8ee2/volumes"
Oct 29 22:49:02.132888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.132812    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13104ade-e48c-489a-8d08-254b7f211239-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13104ade-e48c-489a-8d08-254b7f211239-maas-api-openshift-ca\") pod \"13104ade-e48c-489a-8d08-254b7f211239\" (UID: \"13104ade-e48c-489a-8d08-254b7f211239\") "
Oct 29 22:49:02.132888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.132882    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13104ade-e48c-489a-8d08-254b7f211239-kube-api-access-f9zm5\" (UniqueName: \"kubernetes.io/projected/13104ade-e48c-489a-8d08-254b7f211239-kube-api-access-f9zm5\") pod \"13104ade-e48c-489a-8d08-254b7f211239\" (UID: \"13104ade-e48c-489a-8d08-254b7f211239\") "
Oct 29 22:49:02.133117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.133107    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13104ade-e48c-489a-8d08-254b7f211239-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13104ade-e48c-489a-8d08-254b7f211239-maas-api-xks-ca\") pod \"13104ade-e48c-489a-8d08-254b7f211239\" (UID: \"13104ade-e48c-489a-8d08-254b7f211239\") "
Oct 29 22:49:02.133302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.133278    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13104ade-e48c-489a-8d08-254b7f211239-maas-api-openshift-ca" pod "13104ade-e48c-489a-8d08-254b7f211239" (UID: "13104ade-e48c-489a-8d08-254b7f211239"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:02.133723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.133700    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13104ade-e48c-489a-8d08-254b7f211239-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:02.135655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.135631    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13104ade-e48c-489a-8d08-254b7f211239-maas-api-xks-ca" pod "13104ade-e48c-489a-8d08-254b7f211239" (UID: "13104ade-e48c-489a-8d08-254b7f211239"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:02.135800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.135780    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13104ade-e48c-489a-8d08-254b7f211239-kube-api-access-f9zm5" pod "13104ade-e48c-489a-8d08-254b7f211239" (UID: "13104ade-e48c-489a-8d08-254b7f211239"). InnerVolumeSpecName "kube-api-access-f9zm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:02.234971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.234939    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13104ade-e48c-489a-8d08-254b7f211239-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:02.234971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.234968    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9zm5\" (UniqueName: \"kubernetes.io/projected/13104ade-e48c-489a-8d08-254b7f211239-kube-api-access-f9zm5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:02.769905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.769864    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz" podUID="41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d" containerName="payload-processing" containerID="cri-o://14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800" gracePeriod=25
Oct 29 22:49:02.780853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.780819    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq" podUID="65f1affd-4912-4bdd-b1f9-5d343068e00f" containerName="payload-pre-processing" containerID="cri-o://6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f" gracePeriod=25
Oct 29 22:49:02.882523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.882492    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:02.886167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.886142    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"
Oct 29 22:49:02.886277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.886181    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5" event={"ID":"13104ade-e48c-489a-8d08-254b7f211239","Type":"ContainerDied","Data":"3c9b19aaaa288cb7ca022c39e87c01bfe2d4acb9d58b293dbdd5059f8c00791b"}
Oct 29 22:49:02.886277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:02.886224    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c9b19aaaa288cb7ca022c39e87c01bfe2d4acb9d58b293dbdd5059f8c00791b"
Oct 29 22:49:03.326793 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.326731    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:49:03.366969 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.366942    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-socket\") pod \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") "
Oct 29 22:49:03.367103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.366995    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-credential-socket\") pod \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") "
Oct 29 22:49:03.367103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.367084    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-certs\") pod \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") "
Oct 29 22:49:03.367245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.367226    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-kube-api-access-zh6mp\" (UniqueName: \"kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-kube-api-access-zh6mp\") pod \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") "
Oct 29 22:49:03.367314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.367294    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istiod-ca-cert\") pod \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") "
Oct 29 22:49:03.367401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.367230    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-socket" pod "98bb03a1-504e-4c1e-8df1-d6af23cd10ed" (UID: "98bb03a1-504e-4c1e-8df1-d6af23cd10ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:03.367401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.367284    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-credential-socket" pod "98bb03a1-504e-4c1e-8df1-d6af23cd10ed" (UID: "98bb03a1-504e-4c1e-8df1-d6af23cd10ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:03.367401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.367364    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-token\" (UniqueName: \"kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-token\") pod \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") "
Oct 29 22:49:03.367562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.367458    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-podinfo\") pod \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") "
Oct 29 22:49:03.367562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.367451    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-certs" pod "98bb03a1-504e-4c1e-8df1-d6af23cd10ed" (UID: "98bb03a1-504e-4c1e-8df1-d6af23cd10ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:03.367562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.367508    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-envoy\") pod \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") "
Oct 29 22:49:03.367562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.367539    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-data\") pod \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\" (UID: \"98bb03a1-504e-4c1e-8df1-d6af23cd10ed\") "
Oct 29 22:49:03.367757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.367607    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istiod-ca-cert" pod "98bb03a1-504e-4c1e-8df1-d6af23cd10ed" (UID: "98bb03a1-504e-4c1e-8df1-d6af23cd10ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:03.367983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.367958    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-data" pod "98bb03a1-504e-4c1e-8df1-d6af23cd10ed" (UID: "98bb03a1-504e-4c1e-8df1-d6af23cd10ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:03.368397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.368372    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:03.368503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.368401    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:03.368503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.368416    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:03.368503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.368431    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:03.368503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.368448    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:03.369679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.369652    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-envoy" pod "98bb03a1-504e-4c1e-8df1-d6af23cd10ed" (UID: "98bb03a1-504e-4c1e-8df1-d6af23cd10ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:03.369789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.369691    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-kube-api-access-zh6mp" pod "98bb03a1-504e-4c1e-8df1-d6af23cd10ed" (UID: "98bb03a1-504e-4c1e-8df1-d6af23cd10ed"). InnerVolumeSpecName "kube-api-access-zh6mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:03.369789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.369754    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-token" pod "98bb03a1-504e-4c1e-8df1-d6af23cd10ed" (UID: "98bb03a1-504e-4c1e-8df1-d6af23cd10ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:03.369922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.369876    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-podinfo" pod "98bb03a1-504e-4c1e-8df1-d6af23cd10ed" (UID: "98bb03a1-504e-4c1e-8df1-d6af23cd10ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:03.469951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.469919    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh6mp\" (UniqueName: \"kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-kube-api-access-zh6mp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:03.469951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.469947    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:03.470125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.469958    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:03.470125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.469966    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98bb03a1-504e-4c1e-8df1-d6af23cd10ed-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:03.655202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.655117    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:03.897560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.897524    4860 generic.go:356] "Generic (PLEG): container finished" podID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerID="9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4" exitCode=0
Oct 29 22:49:03.897982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.897571    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"
Oct 29 22:49:03.897982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.897607    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" event={"ID":"98bb03a1-504e-4c1e-8df1-d6af23cd10ed","Type":"ContainerDied","Data":"9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4"}
Oct 29 22:49:03.897982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.897661    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc" event={"ID":"98bb03a1-504e-4c1e-8df1-d6af23cd10ed","Type":"ContainerDied","Data":"2a6ec0ccda9a38f21dfd82367444b206faee149c2cf0ee2264aad4df669053c1"}
Oct 29 22:49:03.897982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.897687    4860 scope.go:122] "RemoveContainer" containerID="9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4"
Oct 29 22:49:03.911212 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.911193    4860 scope.go:122] "RemoveContainer" containerID="a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de"
Oct 29 22:49:03.922115 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.922095    4860 scope.go:122] "RemoveContainer" containerID="9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4"
Oct 29 22:49:03.922415 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:03.922394    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4\": container with ID starting with 9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4 not found: ID does not exist" containerID="9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4"
Oct 29 22:49:03.922516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.922425    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4"} err="failed to get container status \"9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4\": rpc error: code = NotFound desc = could not find container \"9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4\": container with ID starting with 9abc23467f8393a7bed73351a2621bd1164521783b527a6fd4d50ea66d444aa4 not found: ID does not exist"
Oct 29 22:49:03.922516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.922448    4860 scope.go:122] "RemoveContainer" containerID="a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de"
Oct 29 22:49:03.922751 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:03.922720    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de\": container with ID starting with a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de not found: ID does not exist" containerID="a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de"
Oct 29 22:49:03.922862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.922750    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de"} err="failed to get container status \"a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de\": rpc error: code = NotFound desc = could not find container \"a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de\": container with ID starting with a602380b50bebbb93b764cf9b776afdd1790ede1e170e99fb9f4cd1693b805de not found: ID does not exist"
Oct 29 22:49:03.959623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.959601    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"]
Oct 29 22:49:03.969411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:03.969381    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fee294-openshift-default-644cdf5d45-cf2vc"]
Oct 29 22:49:04.114528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:04.114499    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" path="/var/lib/kubelet/pods/98bb03a1-504e-4c1e-8df1-d6af23cd10ed/volumes"
Oct 29 22:49:04.126469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:04.126440    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg" podUID="4bd45d1b-91fb-44fb-95c4-17e38f9d905f" containerName="payload-pre-processing" containerID="cri-o://0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880" gracePeriod=25
Oct 29 22:49:04.979992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:04.979953    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:05.550534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:05.550501    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:05.673242 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:05.673188    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"]
Oct 29 22:49:05.673454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:05.673199    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:05.682802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:05.682777    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5af18-xxl7r"]
Oct 29 22:49:06.113644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:06.113615    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9" path="/var/lib/kubelet/pods/846f68a1-8fc5-4a3d-b1b6-b65ce70b4fa9/volumes"
Oct 29 22:49:06.197512 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:06.197478    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:06.980523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:06.980484    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:07.101823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:07.101783    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:07.380796 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:07.380705    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:07.887243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:07.887210    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg" podUID="4bd45d1b-91fb-44fb-95c4-17e38f9d905f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:08.837304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:08.837256    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:09.020959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:09.020925    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"]
Oct 29 22:49:09.056766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:09.056734    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"]
Oct 29 22:49:09.184647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:09.184611    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:09.329347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:09.312863    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:09.523512 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:09.523421    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:09.633398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:09.633368    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"]
Oct 29 22:49:09.638446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:09.638422    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c7d192-p7tb4"]
Oct 29 22:49:09.648591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:09.648563    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:09.761007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:09.760964    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:10.114109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:10.114079    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fd8747-b6ca-4dd6-a91e-e03851420f04" path="/var/lib/kubelet/pods/46fd8747-b6ca-4dd6-a91e-e03851420f04/volumes"
Oct 29 22:49:10.674200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:10.674164    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz" podUID="41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:10.674379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:10.674162    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq" podUID="65f1affd-4912-4bdd-b1f9-5d343068e00f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:11.077890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:11.077858    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:11.436181 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:11.436150    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:11.849382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:11.849254    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:12.690062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:12.690031    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:12.828718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:12.828682    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:13.005458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.005379    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"]
Oct 29 22:49:13.005690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.005660    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy" containerID="cri-o://355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea" gracePeriod=30
Oct 29 22:49:13.063927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.063857    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:13.337131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.337013    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:13.465365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.465290    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:13.550552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.550521    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"]
Oct 29 22:49:13.551398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.551378    4860 cpu_manager.go:414] "containerMap: removing container" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy"
Oct 29 22:49:13.551475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.551401    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy"
Oct 29 22:49:13.551475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.551426    4860 cpu_manager.go:414] "containerMap: removing container" podUID="46fd8747-b6ca-4dd6-a91e-e03851420f04" containerName="revoke-keys"
Oct 29 22:49:13.551475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.551431    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fd8747-b6ca-4dd6-a91e-e03851420f04" containerName="revoke-keys"
Oct 29 22:49:13.551475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.551452    4860 cpu_manager.go:414] "containerMap: removing container" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy"
Oct 29 22:49:13.551475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.551457    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy"
Oct 29 22:49:13.551475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.551467    4860 cpu_manager.go:414] "containerMap: removing container" podUID="13104ade-e48c-489a-8d08-254b7f211239" containerName="revoke-keys"
Oct 29 22:49:13.551475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.551472    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13104ade-e48c-489a-8d08-254b7f211239" containerName="revoke-keys"
Oct 29 22:49:13.551699 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.551622    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy"
Oct 29 22:49:13.551699 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.551632    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fd8747-b6ca-4dd6-a91e-e03851420f04" containerName="revoke-keys"
Oct 29 22:49:13.551699 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.551642    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="13104ade-e48c-489a-8d08-254b7f211239" containerName="revoke-keys"
Oct 29 22:49:13.555860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.555836    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:13.572160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.572093    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:49:13.572462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.572212    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:49:13.575138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.575111    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"]
Oct 29 22:49:13.653132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.653051    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l\" (UID: \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:13.653316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.653189    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l\" (UID: \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:13.653316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.653276    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm4g5\" (UniqueName: \"kubernetes.io/projected/5294db5d-b390-4b81-a4a1-8b4fea4b396b-kube-api-access-bm4g5\") pod \"maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l\" (UID: \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:13.755101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.755059    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm4g5\" (UniqueName: \"kubernetes.io/projected/5294db5d-b390-4b81-a4a1-8b4fea4b396b-kube-api-access-bm4g5\") pod \"maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l\" (UID: \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:13.755511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.755302    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l\" (UID: \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:13.755511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.755450    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l\" (UID: \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:13.755982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.755955    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l\" (UID: \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:13.757883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.757852    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l\" (UID: \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:13.775867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.775845    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm4g5\" (UniqueName: \"kubernetes.io/projected/5294db5d-b390-4b81-a4a1-8b4fea4b396b-kube-api-access-bm4g5\") pod \"maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l\" (UID: \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:13.874002 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:13.873977    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:14.021767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:14.021718    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2" podUID="504b8724-2295-43a1-bf0c-877e006bd0ae" containerName="payload-processing" containerID="cri-o://fce5b6537c4649c3d61ec68375865556e2ebacb1ad30bd5843d5aaba89b735f5" gracePeriod=25
Oct 29 22:49:14.057047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:14.056975    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk" podUID="277117dc-f565-40c3-81d2-8bc9fbd49346" containerName="payload-pre-processing" containerID="cri-o://784a756d16d2334492c5a09e1dbe585f04e31a85b2a1433a5295f56c1074b8af" gracePeriod=25
Oct 29 22:49:14.204810 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:49:14.204777    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5294db5d_b390_4b81_a4a1_8b4fea4b396b.slice/crio-c9e91fef7da8191a2f7fb08719b00c061e1296225e6da3abe0831f5ae7035a16 WatchSource:0}: Error finding container c9e91fef7da8191a2f7fb08719b00c061e1296225e6da3abe0831f5ae7035a16: Status 404 returned error can't find the container with id c9e91fef7da8191a2f7fb08719b00c061e1296225e6da3abe0831f5ae7035a16
Oct 29 22:49:14.212073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:14.212044    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"]
Oct 29 22:49:15.024254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:15.024226    4860 generic.go:356] "Generic (PLEG): container finished" podID="5294db5d-b390-4b81-a4a1-8b4fea4b396b" containerID="b3bb6db2065d2f80140c0a62338e609337c6c69e694d025651a14fc749fed2af" exitCode=0
Oct 29 22:49:15.024687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:15.024276    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l" event={"ID":"5294db5d-b390-4b81-a4a1-8b4fea4b396b","Type":"ContainerDied","Data":"b3bb6db2065d2f80140c0a62338e609337c6c69e694d025651a14fc749fed2af"}
Oct 29 22:49:15.024687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:15.024345    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l" event={"ID":"5294db5d-b390-4b81-a4a1-8b4fea4b396b","Type":"ContainerStarted","Data":"c9e91fef7da8191a2f7fb08719b00c061e1296225e6da3abe0831f5ae7035a16"}
Oct 29 22:49:15.283692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:15.283621    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:15.542346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:15.542242    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2" podUID="504b8724-2295-43a1-bf0c-877e006bd0ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:15.542513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:15.542360    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk" podUID="277117dc-f565-40c3-81d2-8bc9fbd49346" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:16.190694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.190666    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:16.226448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.226422    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:16.327123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.327030    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:16.327123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.327081    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5294db5d-b390-4b81-a4a1-8b4fea4b396b-kube-api-access-bm4g5\" (UniqueName: \"kubernetes.io/projected/5294db5d-b390-4b81-a4a1-8b4fea4b396b-kube-api-access-bm4g5\") pod \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\" (UID: \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\") "
Oct 29 22:49:16.327390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.327188    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-openshift-ca\") pod \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\" (UID: \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\") "
Oct 29 22:49:16.327390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.327316    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-xks-ca\") pod \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\" (UID: \"5294db5d-b390-4b81-a4a1-8b4fea4b396b\") "
Oct 29 22:49:16.327581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.327561    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-openshift-ca" pod "5294db5d-b390-4b81-a4a1-8b4fea4b396b" (UID: "5294db5d-b390-4b81-a4a1-8b4fea4b396b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:16.327909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.327891    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:16.329350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.329303    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5294db5d-b390-4b81-a4a1-8b4fea4b396b-kube-api-access-bm4g5" pod "5294db5d-b390-4b81-a4a1-8b4fea4b396b" (UID: "5294db5d-b390-4b81-a4a1-8b4fea4b396b"). InnerVolumeSpecName "kube-api-access-bm4g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:16.329456 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.329441    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-xks-ca" pod "5294db5d-b390-4b81-a4a1-8b4fea4b396b" (UID: "5294db5d-b390-4b81-a4a1-8b4fea4b396b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:16.429473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.429448    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5294db5d-b390-4b81-a4a1-8b4fea4b396b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:16.429473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.429473    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm4g5\" (UniqueName: \"kubernetes.io/projected/5294db5d-b390-4b81-a4a1-8b4fea4b396b-kube-api-access-bm4g5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:16.499129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.498475    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"]
Oct 29 22:49:16.509384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.508973    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"]
Oct 29 22:49:16.514935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.514845    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy" containerID="cri-o://627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc" gracePeriod=30
Oct 29 22:49:16.515710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.515371    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerName="istio-proxy" containerID="cri-o://6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e" gracePeriod=30
Oct 29 22:49:16.554618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:16.554590    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:17.016843 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.016812    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"]
Oct 29 22:49:17.017648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.017632    4860 cpu_manager.go:414] "containerMap: removing container" podUID="5294db5d-b390-4b81-a4a1-8b4fea4b396b" containerName="revoke-keys"
Oct 29 22:49:17.017714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.017649    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5294db5d-b390-4b81-a4a1-8b4fea4b396b" containerName="revoke-keys"
Oct 29 22:49:17.017773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.017764    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5294db5d-b390-4b81-a4a1-8b4fea4b396b" containerName="revoke-keys"
Oct 29 22:49:17.017810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.017780    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="98bb03a1-504e-4c1e-8df1-d6af23cd10ed" containerName="istio-proxy"
Oct 29 22:49:17.021864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.021833    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:17.043740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.043714    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"]
Oct 29 22:49:17.054308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.054283    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"
Oct 29 22:49:17.054582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.054294    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l" event={"ID":"5294db5d-b390-4b81-a4a1-8b4fea4b396b","Type":"ContainerDied","Data":"c9e91fef7da8191a2f7fb08719b00c061e1296225e6da3abe0831f5ae7035a16"}
Oct 29 22:49:17.054650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.054615    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9e91fef7da8191a2f7fb08719b00c061e1296225e6da3abe0831f5ae7035a16"
Oct 29 22:49:17.107712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.107679    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:17.144252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.144217    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57\" (UID: \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:17.144462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.144397    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksx5j\" (UniqueName: \"kubernetes.io/projected/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-kube-api-access-ksx5j\") pod \"maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57\" (UID: \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:17.144462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.144435    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57\" (UID: \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:17.245455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.245423    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksx5j\" (UniqueName: \"kubernetes.io/projected/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-kube-api-access-ksx5j\") pod \"maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57\" (UID: \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:17.245858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.245462    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57\" (UID: \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:17.245858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.245662    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57\" (UID: \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:17.246156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.246133    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57\" (UID: \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:17.248189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.248157    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57\" (UID: \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:17.267115 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.267059    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksx5j\" (UniqueName: \"kubernetes.io/projected/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-kube-api-access-ksx5j\") pod \"maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57\" (UID: \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:17.342551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.342526    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:17.650984 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:49:17.650944    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffe539a_ae7b_4878_b2fd_7c389014bbe0.slice/crio-0da6d04b4c19c463184c755274f1e63c8441ed7ed2b40b1cce525f96838605e8 WatchSource:0}: Error finding container 0da6d04b4c19c463184c755274f1e63c8441ed7ed2b40b1cce525f96838605e8: Status 404 returned error can't find the container with id 0da6d04b4c19c463184c755274f1e63c8441ed7ed2b40b1cce525f96838605e8
Oct 29 22:49:17.655063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.655039    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"]
Oct 29 22:49:17.886857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:17.886824    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg" podUID="4bd45d1b-91fb-44fb-95c4-17e38f9d905f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:18.007937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.007905    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"]
Oct 29 22:49:18.058899 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:18.058855    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:49:18.066550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.066523    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57" event={"ID":"2ffe539a-ae7b-4878-b2fd-7c389014bbe0","Type":"ContainerStarted","Data":"0da6d04b4c19c463184c755274f1e63c8441ed7ed2b40b1cce525f96838605e8"}
Oct 29 22:49:18.163407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.162809    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:18.396670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.396639    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:49:18.479901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.479867    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-podinfo\") pod \"307076b4-eceb-48f7-8ca6-5e11f666c439\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") "
Oct 29 22:49:18.480082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.479982    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-data\") pod \"307076b4-eceb-48f7-8ca6-5e11f666c439\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") "
Oct 29 22:49:18.480082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.480036    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-token\" (UniqueName: \"kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-token\") pod \"307076b4-eceb-48f7-8ca6-5e11f666c439\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") "
Oct 29 22:49:18.480082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.480067    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-envoy\") pod \"307076b4-eceb-48f7-8ca6-5e11f666c439\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") "
Oct 29 22:49:18.480356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.480133    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-credential-socket\") pod \"307076b4-eceb-48f7-8ca6-5e11f666c439\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") "
Oct 29 22:49:18.480356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.480163    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-socket\") pod \"307076b4-eceb-48f7-8ca6-5e11f666c439\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") "
Oct 29 22:49:18.480356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.480200    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-kube-api-access-jb4nd\" (UniqueName: \"kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-kube-api-access-jb4nd\") pod \"307076b4-eceb-48f7-8ca6-5e11f666c439\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") "
Oct 29 22:49:18.480356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.480237    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-certs\") pod \"307076b4-eceb-48f7-8ca6-5e11f666c439\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") "
Oct 29 22:49:18.480600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.480309    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-data" pod "307076b4-eceb-48f7-8ca6-5e11f666c439" (UID: "307076b4-eceb-48f7-8ca6-5e11f666c439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.480600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.480458    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/307076b4-eceb-48f7-8ca6-5e11f666c439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307076b4-eceb-48f7-8ca6-5e11f666c439-istiod-ca-cert\") pod \"307076b4-eceb-48f7-8ca6-5e11f666c439\" (UID: \"307076b4-eceb-48f7-8ca6-5e11f666c439\") "
Oct 29 22:49:18.480719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.480577    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-credential-socket" pod "307076b4-eceb-48f7-8ca6-5e11f666c439" (UID: "307076b4-eceb-48f7-8ca6-5e11f666c439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.480719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.480654    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-socket" pod "307076b4-eceb-48f7-8ca6-5e11f666c439" (UID: "307076b4-eceb-48f7-8ca6-5e11f666c439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.480941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.480920    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307076b4-eceb-48f7-8ca6-5e11f666c439-istiod-ca-cert" pod "307076b4-eceb-48f7-8ca6-5e11f666c439" (UID: "307076b4-eceb-48f7-8ca6-5e11f666c439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:18.481172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.481156    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307076b4-eceb-48f7-8ca6-5e11f666c439-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.481172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.481171    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.481310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.481180    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.481310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.481189    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.481861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.481829    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-certs" pod "307076b4-eceb-48f7-8ca6-5e11f666c439" (UID: "307076b4-eceb-48f7-8ca6-5e11f666c439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.482232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.482210    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-podinfo" pod "307076b4-eceb-48f7-8ca6-5e11f666c439" (UID: "307076b4-eceb-48f7-8ca6-5e11f666c439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:18.483188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.483156    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-token" pod "307076b4-eceb-48f7-8ca6-5e11f666c439" (UID: "307076b4-eceb-48f7-8ca6-5e11f666c439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:18.495732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.495706    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-envoy" pod "307076b4-eceb-48f7-8ca6-5e11f666c439" (UID: "307076b4-eceb-48f7-8ca6-5e11f666c439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:18.496443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.496418    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-kube-api-access-jb4nd" pod "307076b4-eceb-48f7-8ca6-5e11f666c439" (UID: "307076b4-eceb-48f7-8ca6-5e11f666c439"). InnerVolumeSpecName "kube-api-access-jb4nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:18.524158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.524097    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:49:18.528404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.528382    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:49:18.582893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.582843    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.583063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.583049    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.583172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.583162    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.583259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.583248    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb4nd\" (UniqueName: \"kubernetes.io/projected/307076b4-eceb-48f7-8ca6-5e11f666c439-kube-api-access-jb4nd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.583361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.583341    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307076b4-eceb-48f7-8ca6-5e11f666c439-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:18.876154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:18.876125    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:19.077689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.077605    4860 generic.go:356] "Generic (PLEG): container finished" podID="2ffe539a-ae7b-4878-b2fd-7c389014bbe0" containerID="7a055cf03557ac8fce95608a81532977488cf68a2763b8e2e0504d11140ea83f" exitCode=0
Oct 29 22:49:19.077869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.077680    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57" event={"ID":"2ffe539a-ae7b-4878-b2fd-7c389014bbe0","Type":"ContainerDied","Data":"7a055cf03557ac8fce95608a81532977488cf68a2763b8e2e0504d11140ea83f"}
Oct 29 22:49:19.079767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.079742    4860 generic.go:356] "Generic (PLEG): container finished" podID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerID="355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea" exitCode=0
Oct 29 22:49:19.079904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.079809    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"
Oct 29 22:49:19.079904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.079809    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" event={"ID":"307076b4-eceb-48f7-8ca6-5e11f666c439","Type":"ContainerDied","Data":"355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea"}
Oct 29 22:49:19.079904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.079858    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782" event={"ID":"307076b4-eceb-48f7-8ca6-5e11f666c439","Type":"ContainerDied","Data":"c77adf228672dfe81fd8f66cd46a7cc96f6db20f5e474b99d0f83542c72a95aa"}
Oct 29 22:49:19.079904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.079880    4860 scope.go:122] "RemoveContainer" containerID="355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea"
Oct 29 22:49:19.093828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.093809    4860 scope.go:122] "RemoveContainer" containerID="a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe"
Oct 29 22:49:19.109745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.109730    4860 scope.go:122] "RemoveContainer" containerID="355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea"
Oct 29 22:49:19.110038 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:19.110012    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea\": container with ID starting with 355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea not found: ID does not exist" containerID="355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea"
Oct 29 22:49:19.110133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.110043    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea"} err="failed to get container status \"355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea\": rpc error: code = NotFound desc = could not find container \"355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea\": container with ID starting with 355d65993299943b9866a72621d9b4ad4281d79b04d9dcd5f4f6fd635c2561ea not found: ID does not exist"
Oct 29 22:49:19.110133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.110060    4860 scope.go:122] "RemoveContainer" containerID="a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe"
Oct 29 22:49:19.110368 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:19.110345    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe\": container with ID starting with a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe not found: ID does not exist" containerID="a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe"
Oct 29 22:49:19.110425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.110374    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe"} err="failed to get container status \"a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe\": rpc error: code = NotFound desc = could not find container \"a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe\": container with ID starting with a3a31f2ee62d94fd133a5d45b1b8f421bf8b1e159dde8f3aaa1d76999dc348fe not found: ID does not exist"
Oct 29 22:49:19.165145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.165116    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"]
Oct 29 22:49:19.178847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:19.178823    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5af18-openshift-default-58d79f487f-tw782"]
Oct 29 22:49:20.116253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.116211    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" path="/var/lib/kubelet/pods/307076b4-eceb-48f7-8ca6-5e11f666c439/volumes"
Oct 29 22:49:20.259989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.259965    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:20.410683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.410602    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-openshift-ca\") pod \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\" (UID: \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\") "
Oct 29 22:49:20.410849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.410734    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-xks-ca\") pod \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\" (UID: \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\") "
Oct 29 22:49:20.410849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.410766    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-kube-api-access-ksx5j\" (UniqueName: \"kubernetes.io/projected/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-kube-api-access-ksx5j\") pod \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\" (UID: \"2ffe539a-ae7b-4878-b2fd-7c389014bbe0\") "
Oct 29 22:49:20.410975 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.410929    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-openshift-ca" pod "2ffe539a-ae7b-4878-b2fd-7c389014bbe0" (UID: "2ffe539a-ae7b-4878-b2fd-7c389014bbe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:20.411384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.411356    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:20.412891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.412867    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-xks-ca" pod "2ffe539a-ae7b-4878-b2fd-7c389014bbe0" (UID: "2ffe539a-ae7b-4878-b2fd-7c389014bbe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:20.413404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.413380    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-kube-api-access-ksx5j" pod "2ffe539a-ae7b-4878-b2fd-7c389014bbe0" (UID: "2ffe539a-ae7b-4878-b2fd-7c389014bbe0"). InnerVolumeSpecName "kube-api-access-ksx5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:20.442315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.442290    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:20.513157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.513118    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:20.513157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.513159    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksx5j\" (UniqueName: \"kubernetes.io/projected/2ffe539a-ae7b-4878-b2fd-7c389014bbe0-kube-api-access-ksx5j\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:20.674521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.674427    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz" podUID="41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:20.674521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:20.674431    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq" podUID="65f1affd-4912-4bdd-b1f9-5d343068e00f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:21.103669 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:21.103636    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57" event={"ID":"2ffe539a-ae7b-4878-b2fd-7c389014bbe0","Type":"ContainerDied","Data":"0da6d04b4c19c463184c755274f1e63c8441ed7ed2b40b1cce525f96838605e8"}
Oct 29 22:49:21.103669 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:21.103677    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0da6d04b4c19c463184c755274f1e63c8441ed7ed2b40b1cce525f96838605e8"
Oct 29 22:49:21.103925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:21.103684    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"
Oct 29 22:49:21.239162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:21.239131    4860 patch_prober.go:29] interesting pod/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:49:21.239554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:21.239187    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:49:21.930295 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:21.913010    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:49:22.003633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.000404    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:49:22.075692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.075610    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-certs\") pod \"ff9440f5-5b2c-4695-b1b1-366c233a7187\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") "
Oct 29 22:49:22.075692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.075661    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-token\" (UniqueName: \"kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-token\") pod \"ff9440f5-5b2c-4695-b1b1-366c233a7187\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") "
Oct 29 22:49:22.075927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.075711    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-kube-api-access-4gvxb\" (UniqueName: \"kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-kube-api-access-4gvxb\") pod \"ff9440f5-5b2c-4695-b1b1-366c233a7187\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") "
Oct 29 22:49:22.075927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.075787    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff9440f5-5b2c-4695-b1b1-366c233a7187-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff9440f5-5b2c-4695-b1b1-366c233a7187-istiod-ca-cert\") pod \"ff9440f5-5b2c-4695-b1b1-366c233a7187\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") "
Oct 29 22:49:22.075927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.075838    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-podinfo\") pod \"ff9440f5-5b2c-4695-b1b1-366c233a7187\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") "
Oct 29 22:49:22.076085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.075933    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-certs" pod "ff9440f5-5b2c-4695-b1b1-366c233a7187" (UID: "ff9440f5-5b2c-4695-b1b1-366c233a7187"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.076085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.075972    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-credential-socket\") pod \"ff9440f5-5b2c-4695-b1b1-366c233a7187\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") "
Oct 29 22:49:22.076200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.076083    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-data\") pod \"ff9440f5-5b2c-4695-b1b1-366c233a7187\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") "
Oct 29 22:49:22.076200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.076139    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-envoy\") pod \"ff9440f5-5b2c-4695-b1b1-366c233a7187\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") "
Oct 29 22:49:22.076302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.076258    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9440f5-5b2c-4695-b1b1-366c233a7187-istiod-ca-cert" pod "ff9440f5-5b2c-4695-b1b1-366c233a7187" (UID: "ff9440f5-5b2c-4695-b1b1-366c233a7187"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:22.076302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.076288    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-socket\") pod \"ff9440f5-5b2c-4695-b1b1-366c233a7187\" (UID: \"ff9440f5-5b2c-4695-b1b1-366c233a7187\") "
Oct 29 22:49:22.076451 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.076424    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-credential-socket" pod "ff9440f5-5b2c-4695-b1b1-366c233a7187" (UID: "ff9440f5-5b2c-4695-b1b1-366c233a7187"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.076597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.076521    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-data" pod "ff9440f5-5b2c-4695-b1b1-366c233a7187" (UID: "ff9440f5-5b2c-4695-b1b1-366c233a7187"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.076806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.076718    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-socket" pod "ff9440f5-5b2c-4695-b1b1-366c233a7187" (UID: "ff9440f5-5b2c-4695-b1b1-366c233a7187"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.076999 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.076980    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.077073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.077005    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.077073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.077018    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.077073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.077032    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff9440f5-5b2c-4695-b1b1-366c233a7187-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.077073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.077045    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.078905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.078870    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-envoy" pod "ff9440f5-5b2c-4695-b1b1-366c233a7187" (UID: "ff9440f5-5b2c-4695-b1b1-366c233a7187"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.079043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.079022    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-podinfo" pod "ff9440f5-5b2c-4695-b1b1-366c233a7187" (UID: "ff9440f5-5b2c-4695-b1b1-366c233a7187"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:22.082105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.082075    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-kube-api-access-4gvxb" pod "ff9440f5-5b2c-4695-b1b1-366c233a7187" (UID: "ff9440f5-5b2c-4695-b1b1-366c233a7187"). InnerVolumeSpecName "kube-api-access-4gvxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:22.084661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.084629    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-token" pod "ff9440f5-5b2c-4695-b1b1-366c233a7187" (UID: "ff9440f5-5b2c-4695-b1b1-366c233a7187"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:22.116954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.116923    4860 generic.go:356] "Generic (PLEG): container finished" podID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerID="6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e" exitCode=0
Oct 29 22:49:22.117081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.117063    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"
Oct 29 22:49:22.118824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.118797    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" event={"ID":"ff9440f5-5b2c-4695-b1b1-366c233a7187","Type":"ContainerDied","Data":"6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e"}
Oct 29 22:49:22.118981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.118901    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4" event={"ID":"ff9440f5-5b2c-4695-b1b1-366c233a7187","Type":"ContainerDied","Data":"9226d0c812688d3613dddbbc35b9aac5eb04e6c40b0429122257e672db0c52ae"}
Oct 29 22:49:22.118981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.118936    4860 scope.go:122] "RemoveContainer" containerID="6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e"
Oct 29 22:49:22.119140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.119093    4860 generic.go:356] "Generic (PLEG): container finished" podID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerID="627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc" exitCode=0
Oct 29 22:49:22.119576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.119416    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" event={"ID":"dffef942-b70f-496b-9dd1-aa8921e066ad","Type":"ContainerDied","Data":"627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc"}
Oct 29 22:49:22.119576 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.119521    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj" event={"ID":"dffef942-b70f-496b-9dd1-aa8921e066ad","Type":"ContainerDied","Data":"20de642a4d9b690563ecc1d2e2e2286fa39ecdbd4d20a9484e5e73d3cbb00873"}
Oct 29 22:49:22.119832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.119807    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"
Oct 29 22:49:22.132340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.132300    4860 scope.go:122] "RemoveContainer" containerID="6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e"
Oct 29 22:49:22.132603 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:22.132584    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e\": container with ID starting with 6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e not found: ID does not exist" containerID="6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e"
Oct 29 22:49:22.132662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.132610    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e"} err="failed to get container status \"6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e\": rpc error: code = NotFound desc = could not find container \"6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e\": container with ID starting with 6729ab58db1d661938dacad1195b4a6c2ef2a8be6d12f27d9cbcaf1d9e1a2a2e not found: ID does not exist"
Oct 29 22:49:22.132662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.132629    4860 scope.go:122] "RemoveContainer" containerID="627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc"
Oct 29 22:49:22.144965 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.144942    4860 scope.go:122] "RemoveContainer" containerID="ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac"
Oct 29 22:49:22.155662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.155645    4860 scope.go:122] "RemoveContainer" containerID="627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc"
Oct 29 22:49:22.155940 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:22.155922    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc\": container with ID starting with 627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc not found: ID does not exist" containerID="627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc"
Oct 29 22:49:22.155993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.155948    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc"} err="failed to get container status \"627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc\": rpc error: code = NotFound desc = could not find container \"627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc\": container with ID starting with 627b4c53829ed7b4edd4b5b6767000c6f4f35c07a3b0ae83d928a41ad7408dbc not found: ID does not exist"
Oct 29 22:49:22.155993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.155965    4860 scope.go:122] "RemoveContainer" containerID="ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac"
Oct 29 22:49:22.156194 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:22.156178    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac\": container with ID starting with ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac not found: ID does not exist" containerID="ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac"
Oct 29 22:49:22.156255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.156200    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac"} err="failed to get container status \"ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac\": rpc error: code = NotFound desc = could not find container \"ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac\": container with ID starting with ba0918d9821a51a454c38f6ea1bb6e968937a13540b9e06a666dbefa41c4f5ac not found: ID does not exist"
Oct 29 22:49:22.176207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.176182    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"]
Oct 29 22:49:22.178098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.178078    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-kube-api-access-xn5t2\" (UniqueName: \"kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-kube-api-access-xn5t2\") pod \"dffef942-b70f-496b-9dd1-aa8921e066ad\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") "
Oct 29 22:49:22.178283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.178268    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-socket\") pod \"dffef942-b70f-496b-9dd1-aa8921e066ad\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") "
Oct 29 22:49:22.178380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.178295    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-data\") pod \"dffef942-b70f-496b-9dd1-aa8921e066ad\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") "
Oct 29 22:49:22.178380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.178370    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-credential-socket\") pod \"dffef942-b70f-496b-9dd1-aa8921e066ad\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") "
Oct 29 22:49:22.178531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.178409    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-podinfo\") pod \"dffef942-b70f-496b-9dd1-aa8921e066ad\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") "
Oct 29 22:49:22.178531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.178441    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-envoy\") pod \"dffef942-b70f-496b-9dd1-aa8921e066ad\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") "
Oct 29 22:49:22.178531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.178485    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-token\" (UniqueName: \"kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-token\") pod \"dffef942-b70f-496b-9dd1-aa8921e066ad\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") "
Oct 29 22:49:22.178700 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.178648    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dffef942-b70f-496b-9dd1-aa8921e066ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffef942-b70f-496b-9dd1-aa8921e066ad-istiod-ca-cert\") pod \"dffef942-b70f-496b-9dd1-aa8921e066ad\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") "
Oct 29 22:49:22.178767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.178735    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-certs\") pod \"dffef942-b70f-496b-9dd1-aa8921e066ad\" (UID: \"dffef942-b70f-496b-9dd1-aa8921e066ad\") "
Oct 29 22:49:22.179869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.179218    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-credential-socket" pod "dffef942-b70f-496b-9dd1-aa8921e066ad" (UID: "dffef942-b70f-496b-9dd1-aa8921e066ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.179869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.179457    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-socket" pod "dffef942-b70f-496b-9dd1-aa8921e066ad" (UID: "dffef942-b70f-496b-9dd1-aa8921e066ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.179869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.179497    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.179869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.179517    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.179869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.179531    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.179869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.179544    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.179869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.179557    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gvxb\" (UniqueName: \"kubernetes.io/projected/ff9440f5-5b2c-4695-b1b1-366c233a7187-kube-api-access-4gvxb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.179869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.179679    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-certs" pod "dffef942-b70f-496b-9dd1-aa8921e066ad" (UID: "dffef942-b70f-496b-9dd1-aa8921e066ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.179869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.179741    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffef942-b70f-496b-9dd1-aa8921e066ad-istiod-ca-cert" pod "dffef942-b70f-496b-9dd1-aa8921e066ad" (UID: "dffef942-b70f-496b-9dd1-aa8921e066ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:22.180589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.180413    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-data" pod "dffef942-b70f-496b-9dd1-aa8921e066ad" (UID: "dffef942-b70f-496b-9dd1-aa8921e066ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.183790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.183741    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-envoy" pod "dffef942-b70f-496b-9dd1-aa8921e066ad" (UID: "dffef942-b70f-496b-9dd1-aa8921e066ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:22.186463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.186439    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-token" pod "dffef942-b70f-496b-9dd1-aa8921e066ad" (UID: "dffef942-b70f-496b-9dd1-aa8921e066ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:22.188757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.188715    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-kube-api-access-xn5t2" pod "dffef942-b70f-496b-9dd1-aa8921e066ad" (UID: "dffef942-b70f-496b-9dd1-aa8921e066ad"). InnerVolumeSpecName "kube-api-access-xn5t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:22.189884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.189860    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-podinfo" pod "dffef942-b70f-496b-9dd1-aa8921e066ad" (UID: "dffef942-b70f-496b-9dd1-aa8921e066ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:22.190015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.189987    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-5gpj4"]
Oct 29 22:49:22.202825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.202790    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:22.281720 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.281677    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.281720 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.281720    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.282189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.281734    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.282189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.281747    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.282189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.281760    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.282189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.281774    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dffef942-b70f-496b-9dd1-aa8921e066ad-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.282189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.281787    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dffef942-b70f-496b-9dd1-aa8921e066ad-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.282189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.281800    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn5t2\" (UniqueName: \"kubernetes.io/projected/dffef942-b70f-496b-9dd1-aa8921e066ad-kube-api-access-xn5t2\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:22.472555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.472527    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"]
Oct 29 22:49:22.477630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:22.477607    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c7d192-openshift-default-c4f554bb7-rgzsj"]
Oct 29 22:49:23.094491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.094403    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"
Oct 29 22:49:23.115296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.115275    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"
Oct 29 22:49:23.129360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.129307    4860 generic.go:356] "Generic (PLEG): container finished" podID="41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d" containerID="14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800" exitCode=0
Oct 29 22:49:23.129440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.129359    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"
Oct 29 22:49:23.129440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.129398    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz" event={"ID":"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d","Type":"ContainerDied","Data":"14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800"}
Oct 29 22:49:23.129532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.129450    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz" event={"ID":"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d","Type":"ContainerDied","Data":"28a32f4408d94288654b22346fa758137996305af129ab2288872be3b9fff572"}
Oct 29 22:49:23.129532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.129476    4860 scope.go:122] "RemoveContainer" containerID="14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800"
Oct 29 22:49:23.130888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.130860    4860 generic.go:356] "Generic (PLEG): container finished" podID="65f1affd-4912-4bdd-b1f9-5d343068e00f" containerID="6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f" exitCode=0
Oct 29 22:49:23.130961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.130905    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"
Oct 29 22:49:23.131011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.130951    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq" event={"ID":"65f1affd-4912-4bdd-b1f9-5d343068e00f","Type":"ContainerDied","Data":"6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f"}
Oct 29 22:49:23.131011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.131002    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq" event={"ID":"65f1affd-4912-4bdd-b1f9-5d343068e00f","Type":"ContainerDied","Data":"1f809153850c527b72976e702816de63e312304d44829537500c50628ade6c5c"}
Oct 29 22:49:23.142009 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.141936    4860 scope.go:122] "RemoveContainer" containerID="14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800"
Oct 29 22:49:23.142262 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:23.142238    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800\": container with ID starting with 14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800 not found: ID does not exist" containerID="14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800"
Oct 29 22:49:23.142357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.142274    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800"} err="failed to get container status \"14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800\": rpc error: code = NotFound desc = could not find container \"14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800\": container with ID starting with 14e7ca9b2dc43b3e1c3f0c83b2dfbc5f82eeda264b1de1c6019a2ee470caf800 not found: ID does not exist"
Oct 29 22:49:23.142357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.142298    4860 scope.go:122] "RemoveContainer" containerID="6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f"
Oct 29 22:49:23.154717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.154696    4860 scope.go:122] "RemoveContainer" containerID="6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f"
Oct 29 22:49:23.155030 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:23.154998    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f\": container with ID starting with 6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f not found: ID does not exist" containerID="6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f"
Oct 29 22:49:23.155128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.155027    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f"} err="failed to get container status \"6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f\": rpc error: code = NotFound desc = could not find container \"6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f\": container with ID starting with 6467bd67422364374a0284135b0d590af08443a72b2add0cc3fadf06be82a70f not found: ID does not exist"
Oct 29 22:49:23.194376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.194355    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-kube-api-access-4f8sh\" (UniqueName: \"kubernetes.io/projected/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-kube-api-access-4f8sh\") pod \"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d\" (UID: \"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d\") "
Oct 29 22:49:23.194493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.194395    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-plugins-config-volume\") pod \"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d\" (UID: \"41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d\") "
Oct 29 22:49:23.194822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.194801    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-plugins-config-volume" pod "41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d" (UID: "41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:23.194953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.194936    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:23.196645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.196621    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-kube-api-access-4f8sh" pod "41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d" (UID: "41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d"). InnerVolumeSpecName "kube-api-access-4f8sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:23.296443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.296362    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f1affd-4912-4bdd-b1f9-5d343068e00f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f1affd-4912-4bdd-b1f9-5d343068e00f-plugins-config-volume\") pod \"65f1affd-4912-4bdd-b1f9-5d343068e00f\" (UID: \"65f1affd-4912-4bdd-b1f9-5d343068e00f\") "
Oct 29 22:49:23.296443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.296406    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f1affd-4912-4bdd-b1f9-5d343068e00f-kube-api-access-tlpt9\" (UniqueName: \"kubernetes.io/projected/65f1affd-4912-4bdd-b1f9-5d343068e00f-kube-api-access-tlpt9\") pod \"65f1affd-4912-4bdd-b1f9-5d343068e00f\" (UID: \"65f1affd-4912-4bdd-b1f9-5d343068e00f\") "
Oct 29 22:49:23.296826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.296716    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f1affd-4912-4bdd-b1f9-5d343068e00f-plugins-config-volume" pod "65f1affd-4912-4bdd-b1f9-5d343068e00f" (UID: "65f1affd-4912-4bdd-b1f9-5d343068e00f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:23.296944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.296930    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f8sh\" (UniqueName: \"kubernetes.io/projected/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d-kube-api-access-4f8sh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:23.296988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.296947    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f1affd-4912-4bdd-b1f9-5d343068e00f-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:23.298596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.298569    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f1affd-4912-4bdd-b1f9-5d343068e00f-kube-api-access-tlpt9" pod "65f1affd-4912-4bdd-b1f9-5d343068e00f" (UID: "65f1affd-4912-4bdd-b1f9-5d343068e00f"). InnerVolumeSpecName "kube-api-access-tlpt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:23.398386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.398362    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlpt9\" (UniqueName: \"kubernetes.io/projected/65f1affd-4912-4bdd-b1f9-5d343068e00f-kube-api-access-tlpt9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:23.484763 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.484732    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"]
Oct 29 22:49:23.489809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.489784    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5af18-5786f6f7f6-v8znz"]
Oct 29 22:49:23.524383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.524356    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"]
Oct 29 22:49:23.528812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.528788    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:23.531557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.531532    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5af18-84d9b694d7-lzhcq"]
Oct 29 22:49:23.944923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:23.944884    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:24.005126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.005090    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"]
Oct 29 22:49:24.035654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.035622    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"]
Oct 29 22:49:24.041302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.041274    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-528518e3-w8fqv"]
Oct 29 22:49:24.046407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.046380    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"]
Oct 29 22:49:24.118833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.118795    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d" path="/var/lib/kubelet/pods/41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d/volumes"
Oct 29 22:49:24.119391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.119367    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50578f72-af6a-4e68-88a8-8854f88ee53f" path="/var/lib/kubelet/pods/50578f72-af6a-4e68-88a8-8854f88ee53f/volumes"
Oct 29 22:49:24.119851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.119830    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f1affd-4912-4bdd-b1f9-5d343068e00f" path="/var/lib/kubelet/pods/65f1affd-4912-4bdd-b1f9-5d343068e00f/volumes"
Oct 29 22:49:24.120714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.120692    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" path="/var/lib/kubelet/pods/dffef942-b70f-496b-9dd1-aa8921e066ad/volumes"
Oct 29 22:49:24.121312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.121291    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" path="/var/lib/kubelet/pods/ff9440f5-5b2c-4695-b1b1-366c233a7187/volumes"
Oct 29 22:49:24.179977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.179937    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:24.314513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.314455    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:24.446692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.446670    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"
Oct 29 22:49:24.619176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.619095    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-kube-api-access-smmgt\" (UniqueName: \"kubernetes.io/projected/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-kube-api-access-smmgt\") pod \"4bd45d1b-91fb-44fb-95c4-17e38f9d905f\" (UID: \"4bd45d1b-91fb-44fb-95c4-17e38f9d905f\") "
Oct 29 22:49:24.619176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.619164    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-plugins-config-volume\") pod \"4bd45d1b-91fb-44fb-95c4-17e38f9d905f\" (UID: \"4bd45d1b-91fb-44fb-95c4-17e38f9d905f\") "
Oct 29 22:49:24.619557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.619533    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-plugins-config-volume" pod "4bd45d1b-91fb-44fb-95c4-17e38f9d905f" (UID: "4bd45d1b-91fb-44fb-95c4-17e38f9d905f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:24.619740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.619725    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:24.621384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.621339    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-kube-api-access-smmgt" pod "4bd45d1b-91fb-44fb-95c4-17e38f9d905f" (UID: "4bd45d1b-91fb-44fb-95c4-17e38f9d905f"). InnerVolumeSpecName "kube-api-access-smmgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:24.731362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:24.730736    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smmgt\" (UniqueName: \"kubernetes.io/projected/4bd45d1b-91fb-44fb-95c4-17e38f9d905f-kube-api-access-smmgt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:25.163147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.163114    4860 generic.go:356] "Generic (PLEG): container finished" podID="4bd45d1b-91fb-44fb-95c4-17e38f9d905f" containerID="0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880" exitCode=0
Oct 29 22:49:25.163384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.163154    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"
Oct 29 22:49:25.163384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.163200    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg" event={"ID":"4bd45d1b-91fb-44fb-95c4-17e38f9d905f","Type":"ContainerDied","Data":"0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880"}
Oct 29 22:49:25.163384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.163254    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg" event={"ID":"4bd45d1b-91fb-44fb-95c4-17e38f9d905f","Type":"ContainerDied","Data":"9253377fa5f7c788af2094252952ce5a0c6a3d3d2c7b1d368816904b6b559433"}
Oct 29 22:49:25.163384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.163283    4860 scope.go:122] "RemoveContainer" containerID="0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880"
Oct 29 22:49:25.175105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.175083    4860 scope.go:122] "RemoveContainer" containerID="0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880"
Oct 29 22:49:25.175411 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:25.175388    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880\": container with ID starting with 0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880 not found: ID does not exist" containerID="0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880"
Oct 29 22:49:25.175479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.175419    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880"} err="failed to get container status \"0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880\": rpc error: code = NotFound desc = could not find container \"0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880\": container with ID starting with 0b657cfeb190ea2fbde2692d63dd4ed959adde9e547e366977b255d5e3342880 not found: ID does not exist"
Oct 29 22:49:25.221375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.221351    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"]
Oct 29 22:49:25.235112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.235089    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5a69c9-95cc9866c-bsqgg"]
Oct 29 22:49:25.521707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.521674    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:25.541875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.541845    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk" podUID="277117dc-f565-40c3-81d2-8bc9fbd49346" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:25.541875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.541867    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2" podUID="504b8724-2295-43a1-bf0c-877e006bd0ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:25.635178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.635145    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:25.956173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:25.956143    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:26.113709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:26.113680    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd45d1b-91fb-44fb-95c4-17e38f9d905f" path="/var/lib/kubelet/pods/4bd45d1b-91fb-44fb-95c4-17e38f9d905f/volumes"
Oct 29 22:49:26.435068 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:26.435012    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:26.617350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:26.612812    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:27.006419 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:27.006380    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:27.564759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:27.564725    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:27.749288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:27.749257    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"]
Oct 29 22:49:27.762039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:27.762007    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5a69c9-rl25z"]
Oct 29 22:49:27.796669 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:27.796637    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:27.928399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:27.928276    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:28.114343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:28.114299    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f07d524-ac1c-47cd-ab07-105ebc7eec0b" path="/var/lib/kubelet/pods/7f07d524-ac1c-47cd-ab07-105ebc7eec0b/volumes"
Oct 29 22:49:28.234508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:28.234474    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:29.007448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.007393    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq" podUID="175c18ae-262c-4dcc-9b92-e0c683837697" containerName="payload-processing" containerID="cri-o://dc0f80074b668b5c077bb907e34606311950c7ff1d99164bafda28a0a6587c40" gracePeriod=25
Oct 29 22:49:29.047423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.047376    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm" podUID="a5000257-c412-436d-92f6-9b31c531e94b" containerName="payload-pre-processing" containerID="cri-o://01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15" gracePeriod=25
Oct 29 22:49:29.264627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.264547    4860 generic.go:356] "Generic (PLEG): container finished" podID="175c18ae-262c-4dcc-9b92-e0c683837697" containerID="dc0f80074b668b5c077bb907e34606311950c7ff1d99164bafda28a0a6587c40" exitCode=0
Oct 29 22:49:29.265121 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.265084    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq" event={"ID":"175c18ae-262c-4dcc-9b92-e0c683837697","Type":"ContainerDied","Data":"dc0f80074b668b5c077bb907e34606311950c7ff1d99164bafda28a0a6587c40"}
Oct 29 22:49:29.326559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.326535    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"
Oct 29 22:49:29.334919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.334887    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175c18ae-262c-4dcc-9b92-e0c683837697-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/175c18ae-262c-4dcc-9b92-e0c683837697-plugins-config-volume\") pod \"175c18ae-262c-4dcc-9b92-e0c683837697\" (UID: \"175c18ae-262c-4dcc-9b92-e0c683837697\") "
Oct 29 22:49:29.335150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.335131    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175c18ae-262c-4dcc-9b92-e0c683837697-kube-api-access-j6k9z\" (UniqueName: \"kubernetes.io/projected/175c18ae-262c-4dcc-9b92-e0c683837697-kube-api-access-j6k9z\") pod \"175c18ae-262c-4dcc-9b92-e0c683837697\" (UID: \"175c18ae-262c-4dcc-9b92-e0c683837697\") "
Oct 29 22:49:29.335305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.335281    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175c18ae-262c-4dcc-9b92-e0c683837697-plugins-config-volume" pod "175c18ae-262c-4dcc-9b92-e0c683837697" (UID: "175c18ae-262c-4dcc-9b92-e0c683837697"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:29.335747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.335727    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/175c18ae-262c-4dcc-9b92-e0c683837697-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:29.337652 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.337625    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175c18ae-262c-4dcc-9b92-e0c683837697-kube-api-access-j6k9z" pod "175c18ae-262c-4dcc-9b92-e0c683837697" (UID: "175c18ae-262c-4dcc-9b92-e0c683837697"). InnerVolumeSpecName "kube-api-access-j6k9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:29.438104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.438068    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6k9z\" (UniqueName: \"kubernetes.io/projected/175c18ae-262c-4dcc-9b92-e0c683837697-kube-api-access-j6k9z\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:29.473539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.473492    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:29.602338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:29.602235    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:30.275713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:30.275683    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"
Oct 29 22:49:30.276216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:30.275714    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq" event={"ID":"175c18ae-262c-4dcc-9b92-e0c683837697","Type":"ContainerDied","Data":"e3f56fd32fcee180de98ca0fb18d1041c02d2f9820764e4531575a1c9cd2baa7"}
Oct 29 22:49:30.276216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:30.275773    4860 scope.go:122] "RemoveContainer" containerID="dc0f80074b668b5c077bb907e34606311950c7ff1d99164bafda28a0a6587c40"
Oct 29 22:49:30.335274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:30.335248    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"]
Oct 29 22:49:30.347979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:30.347954    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e619b6-7f44c56884-ctrpq"]
Oct 29 22:49:31.454640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:31.454608    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:31.600150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:31.600070    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:31.720432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:31.720349    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:32.114188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:32.114113    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175c18ae-262c-4dcc-9b92-e0c683837697" path="/var/lib/kubelet/pods/175c18ae-262c-4dcc-9b92-e0c683837697/volumes"
Oct 29 22:49:33.032036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.032000    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb" podUID="f6d6c31a-f447-4e47-9efa-4de8ea630a64" containerName="main" containerID="cri-o://3cab11e6a7627b32ef3e8c9cd4143658c3a9500b9c0a29eb592b2eeae23beb7d" gracePeriod=45
Oct 29 22:49:33.177994 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.177953    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:33.320315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.320272    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:33.320801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.320772    4860 generic.go:356] "Generic (PLEG): container finished" podID="f6d6c31a-f447-4e47-9efa-4de8ea630a64" containerID="3cab11e6a7627b32ef3e8c9cd4143658c3a9500b9c0a29eb592b2eeae23beb7d" exitCode=0
Oct 29 22:49:33.321088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.321067    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb" event={"ID":"f6d6c31a-f447-4e47-9efa-4de8ea630a64","Type":"ContainerDied","Data":"3cab11e6a7627b32ef3e8c9cd4143658c3a9500b9c0a29eb592b2eeae23beb7d"}
Oct 29 22:49:33.321235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.321215    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb" event={"ID":"f6d6c31a-f447-4e47-9efa-4de8ea630a64","Type":"ContainerDied","Data":"db0b90633c42a54cd989816d79022c2ccd28c9abf48e50061760fe9be6803762"}
Oct 29 22:49:33.321370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.321354    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db0b90633c42a54cd989816d79022c2ccd28c9abf48e50061760fe9be6803762"
Oct 29 22:49:33.324824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.324800    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:49:33.331789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.331755    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-dshm\") pod \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") "
Oct 29 22:49:33.331907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.331865    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tls-certs\") pod \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") "
Oct 29 22:49:33.331976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.331908    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tmp-dir\") pod \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") "
Oct 29 22:49:33.331976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.331960    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d6c31a-f447-4e47-9efa-4de8ea630a64-kube-api-access-gmz9n\" (UniqueName: \"kubernetes.io/projected/f6d6c31a-f447-4e47-9efa-4de8ea630a64-kube-api-access-gmz9n\") pod \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") "
Oct 29 22:49:33.332091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.332000    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-home\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-home\") pod \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") "
Oct 29 22:49:33.332091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.332020    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-model-cache\") pod \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\" (UID: \"f6d6c31a-f447-4e47-9efa-4de8ea630a64\") "
Oct 29 22:49:33.332244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.332216    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tmp-dir" pod "f6d6c31a-f447-4e47-9efa-4de8ea630a64" (UID: "f6d6c31a-f447-4e47-9efa-4de8ea630a64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:33.332395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.332370    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-home" pod "f6d6c31a-f447-4e47-9efa-4de8ea630a64" (UID: "f6d6c31a-f447-4e47-9efa-4de8ea630a64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:33.332467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.332390    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-model-cache" pod "f6d6c31a-f447-4e47-9efa-4de8ea630a64" (UID: "f6d6c31a-f447-4e47-9efa-4de8ea630a64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:33.332631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.332607    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:33.332631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.332632    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:33.332784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.332645    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:33.334278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.334254    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tls-certs" pod "f6d6c31a-f447-4e47-9efa-4de8ea630a64" (UID: "f6d6c31a-f447-4e47-9efa-4de8ea630a64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:33.334386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.334260    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-dshm" pod "f6d6c31a-f447-4e47-9efa-4de8ea630a64" (UID: "f6d6c31a-f447-4e47-9efa-4de8ea630a64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:33.334386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.334375    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d6c31a-f447-4e47-9efa-4de8ea630a64-kube-api-access-gmz9n" pod "f6d6c31a-f447-4e47-9efa-4de8ea630a64" (UID: "f6d6c31a-f447-4e47-9efa-4de8ea630a64"). InnerVolumeSpecName "kube-api-access-gmz9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:33.465824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.444264    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d6c31a-f447-4e47-9efa-4de8ea630a64-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:33.465824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.444298    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d6c31a-f447-4e47-9efa-4de8ea630a64-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:33.465824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.444313    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmz9n\" (UniqueName: \"kubernetes.io/projected/f6d6c31a-f447-4e47-9efa-4de8ea630a64-kube-api-access-gmz9n\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:33.532644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:33.532535    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:34.339760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.339544    4860 generic.go:356] "Generic (PLEG): container finished" podID="277117dc-f565-40c3-81d2-8bc9fbd49346" containerID="784a756d16d2334492c5a09e1dbe585f04e31a85b2a1433a5295f56c1074b8af" exitCode=0
Oct 29 22:49:34.339760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.339625    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk" event={"ID":"277117dc-f565-40c3-81d2-8bc9fbd49346","Type":"ContainerDied","Data":"784a756d16d2334492c5a09e1dbe585f04e31a85b2a1433a5295f56c1074b8af"}
Oct 29 22:49:34.342981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.342955    4860 generic.go:356] "Generic (PLEG): container finished" podID="504b8724-2295-43a1-bf0c-877e006bd0ae" containerID="fce5b6537c4649c3d61ec68375865556e2ebacb1ad30bd5843d5aaba89b735f5" exitCode=0
Oct 29 22:49:34.343216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.343203    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"
Oct 29 22:49:34.343432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.343407    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2" event={"ID":"504b8724-2295-43a1-bf0c-877e006bd0ae","Type":"ContainerDied","Data":"fce5b6537c4649c3d61ec68375865556e2ebacb1ad30bd5843d5aaba89b735f5"}
Oct 29 22:49:34.373046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.373028    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"
Oct 29 22:49:34.411609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.411585    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"]
Oct 29 22:49:34.419747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.419727    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"
Oct 29 22:49:34.424427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.424407    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-6fc6bd/auth-test-model-6fc6bd-kserve-7f89dcd945-cnxdb"]
Oct 29 22:49:34.458570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.458538    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/504b8724-2295-43a1-bf0c-877e006bd0ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504b8724-2295-43a1-bf0c-877e006bd0ae-plugins-config-volume\") pod \"504b8724-2295-43a1-bf0c-877e006bd0ae\" (UID: \"504b8724-2295-43a1-bf0c-877e006bd0ae\") "
Oct 29 22:49:34.458704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.458599    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504b8724-2295-43a1-bf0c-877e006bd0ae-kube-api-access-lpm7b\" (UniqueName: \"kubernetes.io/projected/504b8724-2295-43a1-bf0c-877e006bd0ae-kube-api-access-lpm7b\") pod \"504b8724-2295-43a1-bf0c-877e006bd0ae\" (UID: \"504b8724-2295-43a1-bf0c-877e006bd0ae\") "
Oct 29 22:49:34.458704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.458645    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277117dc-f565-40c3-81d2-8bc9fbd49346-kube-api-access-qn8nz\" (UniqueName: \"kubernetes.io/projected/277117dc-f565-40c3-81d2-8bc9fbd49346-kube-api-access-qn8nz\") pod \"277117dc-f565-40c3-81d2-8bc9fbd49346\" (UID: \"277117dc-f565-40c3-81d2-8bc9fbd49346\") "
Oct 29 22:49:34.458850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.458715    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277117dc-f565-40c3-81d2-8bc9fbd49346-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277117dc-f565-40c3-81d2-8bc9fbd49346-plugins-config-volume\") pod \"277117dc-f565-40c3-81d2-8bc9fbd49346\" (UID: \"277117dc-f565-40c3-81d2-8bc9fbd49346\") "
Oct 29 22:49:34.459006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.458976    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504b8724-2295-43a1-bf0c-877e006bd0ae-plugins-config-volume" pod "504b8724-2295-43a1-bf0c-877e006bd0ae" (UID: "504b8724-2295-43a1-bf0c-877e006bd0ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:34.459392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.459367    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277117dc-f565-40c3-81d2-8bc9fbd49346-plugins-config-volume" pod "277117dc-f565-40c3-81d2-8bc9fbd49346" (UID: "277117dc-f565-40c3-81d2-8bc9fbd49346"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:34.459562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.459544    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/504b8724-2295-43a1-bf0c-877e006bd0ae-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:34.459626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.459567    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277117dc-f565-40c3-81d2-8bc9fbd49346-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:34.462103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.462075    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504b8724-2295-43a1-bf0c-877e006bd0ae-kube-api-access-lpm7b" pod "504b8724-2295-43a1-bf0c-877e006bd0ae" (UID: "504b8724-2295-43a1-bf0c-877e006bd0ae"). InnerVolumeSpecName "kube-api-access-lpm7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:34.462394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.462365    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277117dc-f565-40c3-81d2-8bc9fbd49346-kube-api-access-qn8nz" pod "277117dc-f565-40c3-81d2-8bc9fbd49346" (UID: "277117dc-f565-40c3-81d2-8bc9fbd49346"). InnerVolumeSpecName "kube-api-access-qn8nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:34.561110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.561045    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpm7b\" (UniqueName: \"kubernetes.io/projected/504b8724-2295-43a1-bf0c-877e006bd0ae-kube-api-access-lpm7b\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:34.561110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.561071    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn8nz\" (UniqueName: \"kubernetes.io/projected/277117dc-f565-40c3-81d2-8bc9fbd49346-kube-api-access-qn8nz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:34.820577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.820489    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:34.952552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:34.952516    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:35.130130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.130093    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"]
Oct 29 22:49:35.130307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.130282    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d" podUID="087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" containerName="istio-proxy" containerID="cri-o://f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf" gracePeriod=30
Oct 29 22:49:35.187134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.187105    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:35.312631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.312595    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:35.393393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.388850    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"
Oct 29 22:49:35.393393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.388870    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk" event={"ID":"277117dc-f565-40c3-81d2-8bc9fbd49346","Type":"ContainerDied","Data":"21f8d982060b5ef2873582598f2a6f5c0364722bb8ce2172adf82c194809f9b4"}
Oct 29 22:49:35.393393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.388943    4860 scope.go:122] "RemoveContainer" containerID="784a756d16d2334492c5a09e1dbe585f04e31a85b2a1433a5295f56c1074b8af"
Oct 29 22:49:35.393393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.391027    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2" event={"ID":"504b8724-2295-43a1-bf0c-877e006bd0ae","Type":"ContainerDied","Data":"bf8b1fd978e5ca919f2ae5e9c5d64fb4080d083d14acb4b592511b31d6056040"}
Oct 29 22:49:35.393393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.391117    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"
Oct 29 22:49:35.404461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.404435    4860 scope.go:122] "RemoveContainer" containerID="fce5b6537c4649c3d61ec68375865556e2ebacb1ad30bd5843d5aaba89b735f5"
Oct 29 22:49:35.460493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.460458    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"]
Oct 29 22:49:35.468482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.468448    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8147a8-f8bf948b4-575w2"]
Oct 29 22:49:35.503188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.503158    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"]
Oct 29 22:49:35.515423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.515388    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8147a8-7f7ff5cc5-4fxdk"]
Oct 29 22:49:35.656590 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.656513    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"]
Oct 29 22:49:35.657266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657245    4860 cpu_manager.go:414] "containerMap: removing container" podUID="65f1affd-4912-4bdd-b1f9-5d343068e00f" containerName="payload-pre-processing"
Oct 29 22:49:35.657416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657268    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f1affd-4912-4bdd-b1f9-5d343068e00f" containerName="payload-pre-processing"
Oct 29 22:49:35.657416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657289    4860 cpu_manager.go:414] "containerMap: removing container" podUID="504b8724-2295-43a1-bf0c-877e006bd0ae" containerName="payload-processing"
Oct 29 22:49:35.657416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657297    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504b8724-2295-43a1-bf0c-877e006bd0ae" containerName="payload-processing"
Oct 29 22:49:35.657416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657319    4860 cpu_manager.go:414] "containerMap: removing container" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy"
Oct 29 22:49:35.657416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657341    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy"
Oct 29 22:49:35.657416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657359    4860 cpu_manager.go:414] "containerMap: removing container" podUID="41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d" containerName="payload-processing"
Oct 29 22:49:35.657416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657366    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d" containerName="payload-processing"
Oct 29 22:49:35.657416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657376    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4bd45d1b-91fb-44fb-95c4-17e38f9d905f" containerName="payload-pre-processing"
Oct 29 22:49:35.657416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657384    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd45d1b-91fb-44fb-95c4-17e38f9d905f" containerName="payload-pre-processing"
Oct 29 22:49:35.657416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657399    4860 cpu_manager.go:414] "containerMap: removing container" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy"
Oct 29 22:49:35.657416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657406    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657429    4860 cpu_manager.go:414] "containerMap: removing container" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657437    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657460    4860 cpu_manager.go:414] "containerMap: removing container" podUID="277117dc-f565-40c3-81d2-8bc9fbd49346" containerName="payload-pre-processing"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657468    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277117dc-f565-40c3-81d2-8bc9fbd49346" containerName="payload-pre-processing"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657481    4860 cpu_manager.go:414] "containerMap: removing container" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657488    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657503    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerName="istio-proxy"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657509    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerName="istio-proxy"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657531    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2ffe539a-ae7b-4878-b2fd-7c389014bbe0" containerName="revoke-keys"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657538    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ffe539a-ae7b-4878-b2fd-7c389014bbe0" containerName="revoke-keys"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657549    4860 cpu_manager.go:414] "containerMap: removing container" podUID="175c18ae-262c-4dcc-9b92-e0c683837697" containerName="payload-processing"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657556    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175c18ae-262c-4dcc-9b92-e0c683837697" containerName="payload-processing"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657583    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f6d6c31a-f447-4e47-9efa-4de8ea630a64" containerName="main"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657591    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d6c31a-f447-4e47-9efa-4de8ea630a64" containerName="main"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657776    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ffe539a-ae7b-4878-b2fd-7c389014bbe0" containerName="revoke-keys"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657792    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="175c18ae-262c-4dcc-9b92-e0c683837697" containerName="payload-processing"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657807    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d6c31a-f447-4e47-9efa-4de8ea630a64" containerName="main"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657818    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f1affd-4912-4bdd-b1f9-5d343068e00f" containerName="payload-pre-processing"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657829    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="504b8724-2295-43a1-bf0c-877e006bd0ae" containerName="payload-processing"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657842    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657851    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d11cf5-1e1a-43cb-97b7-72a5c7f7d81d" containerName="payload-processing"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657861    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd45d1b-91fb-44fb-95c4-17e38f9d905f" containerName="payload-pre-processing"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657870    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="307076b4-eceb-48f7-8ca6-5e11f666c439" containerName="istio-proxy"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657882    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657891    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="277117dc-f565-40c3-81d2-8bc9fbd49346" containerName="payload-pre-processing"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657902    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffef942-b70f-496b-9dd1-aa8921e066ad" containerName="istio-proxy"
Oct 29 22:49:35.657985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.657912    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff9440f5-5b2c-4695-b1b1-366c233a7187" containerName="istio-proxy"
Oct 29 22:49:35.663591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.663569    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:35.677533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.677489    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf\" (UID: \"9014903f-7eee-4a29-bec9-933b6f83ca47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:35.677666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.677612    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp52c\" (UniqueName: \"kubernetes.io/projected/9014903f-7eee-4a29-bec9-933b6f83ca47-kube-api-access-rp52c\") pod \"maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf\" (UID: \"9014903f-7eee-4a29-bec9-933b6f83ca47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:35.677937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.677915    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf\" (UID: \"9014903f-7eee-4a29-bec9-933b6f83ca47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:35.681474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.681451    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"]
Oct 29 22:49:35.698122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.698100    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:49:35.698261 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.698242    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:49:35.778966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.778943    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf\" (UID: \"9014903f-7eee-4a29-bec9-933b6f83ca47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:35.779093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.779001    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf\" (UID: \"9014903f-7eee-4a29-bec9-933b6f83ca47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:35.779093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.779078    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp52c\" (UniqueName: \"kubernetes.io/projected/9014903f-7eee-4a29-bec9-933b6f83ca47-kube-api-access-rp52c\") pod \"maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf\" (UID: \"9014903f-7eee-4a29-bec9-933b6f83ca47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:35.779698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.779676    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf\" (UID: \"9014903f-7eee-4a29-bec9-933b6f83ca47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:35.781431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.781402    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf\" (UID: \"9014903f-7eee-4a29-bec9-933b6f83ca47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:35.805980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.805956    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp52c\" (UniqueName: \"kubernetes.io/projected/9014903f-7eee-4a29-bec9-933b6f83ca47-kube-api-access-rp52c\") pod \"maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf\" (UID: \"9014903f-7eee-4a29-bec9-933b6f83ca47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:35.979543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:35.979518    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:36.119158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:36.119119    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277117dc-f565-40c3-81d2-8bc9fbd49346" path="/var/lib/kubelet/pods/277117dc-f565-40c3-81d2-8bc9fbd49346/volumes"
Oct 29 22:49:36.119870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:36.119848    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="504b8724-2295-43a1-bf0c-877e006bd0ae" path="/var/lib/kubelet/pods/504b8724-2295-43a1-bf0c-877e006bd0ae/volumes"
Oct 29 22:49:36.120528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:36.120494    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d6c31a-f447-4e47-9efa-4de8ea630a64" path="/var/lib/kubelet/pods/f6d6c31a-f447-4e47-9efa-4de8ea630a64/volumes"
Oct 29 22:49:36.280255 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:49:36.280227    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9014903f_7eee_4a29_bec9_933b6f83ca47.slice/crio-fd982b3867e168b51004046c132587d9a2fe5b8a346cb546e15f6514fa757377 WatchSource:0}: Error finding container fd982b3867e168b51004046c132587d9a2fe5b8a346cb546e15f6514fa757377: Status 404 returned error can't find the container with id fd982b3867e168b51004046c132587d9a2fe5b8a346cb546e15f6514fa757377
Oct 29 22:49:36.287864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:36.287837    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"]
Oct 29 22:49:36.403093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:36.403065    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf" event={"ID":"9014903f-7eee-4a29-bec9-933b6f83ca47","Type":"ContainerStarted","Data":"fd982b3867e168b51004046c132587d9a2fe5b8a346cb546e15f6514fa757377"}
Oct 29 22:49:37.149405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:37.149366    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:37.414114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:37.414019    4860 generic.go:356] "Generic (PLEG): container finished" podID="9014903f-7eee-4a29-bec9-933b6f83ca47" containerID="8510b0a9f7eb93d61cb7da386486c950a777cb80c0e1c97b3f97b9ca45102d1f" exitCode=0
Oct 29 22:49:37.414114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:37.414099    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf" event={"ID":"9014903f-7eee-4a29-bec9-933b6f83ca47","Type":"ContainerDied","Data":"8510b0a9f7eb93d61cb7da386486c950a777cb80c0e1c97b3f97b9ca45102d1f"}
Oct 29 22:49:38.335781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.335732    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm" podUID="a5000257-c412-436d-92f6-9b31c531e94b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:38.632050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.632027    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:38.709746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.709708    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:38.721589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.721559    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9014903f-7eee-4a29-bec9-933b6f83ca47-kube-api-access-rp52c\" (UniqueName: \"kubernetes.io/projected/9014903f-7eee-4a29-bec9-933b6f83ca47-kube-api-access-rp52c\") pod \"9014903f-7eee-4a29-bec9-933b6f83ca47\" (UID: \"9014903f-7eee-4a29-bec9-933b6f83ca47\") "
Oct 29 22:49:38.721720 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.721650    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-openshift-ca\") pod \"9014903f-7eee-4a29-bec9-933b6f83ca47\" (UID: \"9014903f-7eee-4a29-bec9-933b6f83ca47\") "
Oct 29 22:49:38.721826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.721808    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-xks-ca\") pod \"9014903f-7eee-4a29-bec9-933b6f83ca47\" (UID: \"9014903f-7eee-4a29-bec9-933b6f83ca47\") "
Oct 29 22:49:38.722270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.722080    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-openshift-ca" pod "9014903f-7eee-4a29-bec9-933b6f83ca47" (UID: "9014903f-7eee-4a29-bec9-933b6f83ca47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:38.722557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.722538    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:38.724618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.724588    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-xks-ca" pod "9014903f-7eee-4a29-bec9-933b6f83ca47" (UID: "9014903f-7eee-4a29-bec9-933b6f83ca47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:49:38.726762 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.726732    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9014903f-7eee-4a29-bec9-933b6f83ca47-kube-api-access-rp52c" pod "9014903f-7eee-4a29-bec9-933b6f83ca47" (UID: "9014903f-7eee-4a29-bec9-933b6f83ca47"). InnerVolumeSpecName "kube-api-access-rp52c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:38.824483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.824456    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9014903f-7eee-4a29-bec9-933b6f83ca47-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:38.824483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.824482    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp52c\" (UniqueName: \"kubernetes.io/projected/9014903f-7eee-4a29-bec9-933b6f83ca47-kube-api-access-rp52c\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:38.850532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:38.850463    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:39.433359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:39.433306    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"
Oct 29 22:49:39.433359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:39.433319    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf" event={"ID":"9014903f-7eee-4a29-bec9-933b6f83ca47","Type":"ContainerDied","Data":"fd982b3867e168b51004046c132587d9a2fe5b8a346cb546e15f6514fa757377"}
Oct 29 22:49:39.433544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:39.433400    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd982b3867e168b51004046c132587d9a2fe5b8a346cb546e15f6514fa757377"
Oct 29 22:49:40.348527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.348490    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:40.427598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.427574    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:49:40.443255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.443226    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-certs\") pod \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") "
Oct 29 22:49:40.443407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.443375    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-credential-socket\") pod \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") "
Oct 29 22:49:40.443495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.443422    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-envoy\") pod \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") "
Oct 29 22:49:40.443556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.443524    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-kube-api-access-jpglb\" (UniqueName: \"kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-kube-api-access-jpglb\") pod \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") "
Oct 29 22:49:40.443612 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.443574    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istiod-ca-cert\") pod \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") "
Oct 29 22:49:40.443612 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.443572    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-certs" pod "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" (UID: "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.443625    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-socket\") pod \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") "
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.443663    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-credential-socket" pod "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" (UID: "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.443833    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-socket" pod "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" (UID: "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.443955    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istiod-ca-cert" pod "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" (UID: "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.444095    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-token\" (UniqueName: \"kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-token\") pod \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") "
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.444148    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-data\") pod \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") "
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.444180    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-podinfo\") pod \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\" (UID: \"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09\") "
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.444227    4860 generic.go:356] "Generic (PLEG): container finished" podID="087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" containerID="f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf" exitCode=0
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.444274    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d" event={"ID":"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09","Type":"ContainerDied","Data":"f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf"}
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.444341    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d" event={"ID":"087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09","Type":"ContainerDied","Data":"191069df5a66c19ff3d1e412480480de0a7b9cd2799e863c4e468cc71aaf716a"}
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.444368    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.444373    4860 scope.go:122] "RemoveContainer" containerID="f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf"
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.444461    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-data" pod "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" (UID: "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.445068    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.445094    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.445108    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.445121    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.445161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.445134    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.446404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.446379    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-envoy" pod "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" (UID: "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:40.446512 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.446480    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-kube-api-access-jpglb" pod "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" (UID: "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09"). InnerVolumeSpecName "kube-api-access-jpglb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:40.446566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.446540    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-token" pod "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" (UID: "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:40.446614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.446563    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-podinfo" pod "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" (UID: "087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:40.476829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.476806    4860 scope.go:122] "RemoveContainer" containerID="f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf"
Oct 29 22:49:40.477154 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:40.477133    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf\": container with ID starting with f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf not found: ID does not exist" containerID="f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf"
Oct 29 22:49:40.477230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.477164    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf"} err="failed to get container status \"f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf\": rpc error: code = NotFound desc = could not find container \"f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf\": container with ID starting with f958df42d7af6d9a55267b0a263d2491dfd5c9b0caa4f13333504dfa3d67eeaf not found: ID does not exist"
Oct 29 22:49:40.554740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.554644    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.554740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.554680    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpglb\" (UniqueName: \"kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-kube-api-access-jpglb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.554740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.554695    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.554740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.554708    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:40.803642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.803608    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"]
Oct 29 22:49:40.810176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.810138    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5a69c9-openshift-default-cbff5c97-dkq5d"]
Oct 29 22:49:40.940368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:40.940299    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:41.055400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:41.055311    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"]
Oct 29 22:49:41.060186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:41.060155    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8147a8-qk7t5"]
Oct 29 22:49:41.068706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:41.068681    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:41.227992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:41.227963    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:41.572661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:41.572625    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:42.113825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:42.113792    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" path="/var/lib/kubelet/pods/087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09/volumes"
Oct 29 22:49:42.114266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:42.114252    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13104ade-e48c-489a-8d08-254b7f211239" path="/var/lib/kubelet/pods/13104ade-e48c-489a-8d08-254b7f211239/volumes"
Oct 29 22:49:42.760663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:42.760628    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:44.397905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:44.397873    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:45.110055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:45.110020    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:45.830672 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:45.830629    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:45.885715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:45.885680    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"]
Oct 29 22:49:46.016428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:46.016400    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:46.148868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:46.148770    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:46.325147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:46.325113    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:46.428146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:46.428075    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"]
Oct 29 22:49:46.432540 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:46.432515    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e619b6-fdk6l"]
Oct 29 22:49:46.447077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:46.447053    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:46.572924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:46.572892    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:46.948507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:46.948449    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:47.844443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:47.844411    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:47.966617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:47.966579    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:48.059054 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:48.059011    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:49:48.121261 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:48.121175    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5294db5d-b390-4b81-a4a1-8b4fea4b396b" path="/var/lib/kubelet/pods/5294db5d-b390-4b81-a4a1-8b4fea4b396b/volumes"
Oct 29 22:49:48.130981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:48.130947    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"]
Oct 29 22:49:48.131212 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:48.131182    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" podUID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" containerName="istio-proxy" containerID="cri-o://12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a" gracePeriod=30
Oct 29 22:49:48.197437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:48.197406    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:48.340817 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:48.340781    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:48.355312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:48.355273    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm" podUID="a5000257-c412-436d-92f6-9b31c531e94b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:49.325747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.325693    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"
Oct 29 22:49:49.390892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.390852    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5000257-c412-436d-92f6-9b31c531e94b-kube-api-access-66dsq\" (UniqueName: \"kubernetes.io/projected/a5000257-c412-436d-92f6-9b31c531e94b-kube-api-access-66dsq\") pod \"a5000257-c412-436d-92f6-9b31c531e94b\" (UID: \"a5000257-c412-436d-92f6-9b31c531e94b\") "
Oct 29 22:49:49.391056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.390964    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5000257-c412-436d-92f6-9b31c531e94b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5000257-c412-436d-92f6-9b31c531e94b-plugins-config-volume\") pod \"a5000257-c412-436d-92f6-9b31c531e94b\" (UID: \"a5000257-c412-436d-92f6-9b31c531e94b\") "
Oct 29 22:49:49.391545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.391487    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5000257-c412-436d-92f6-9b31c531e94b-plugins-config-volume" pod "a5000257-c412-436d-92f6-9b31c531e94b" (UID: "a5000257-c412-436d-92f6-9b31c531e94b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:49.391745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.391717    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5000257-c412-436d-92f6-9b31c531e94b-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:49.395580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.395553    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5000257-c412-436d-92f6-9b31c531e94b-kube-api-access-66dsq" pod "a5000257-c412-436d-92f6-9b31c531e94b" (UID: "a5000257-c412-436d-92f6-9b31c531e94b"). InnerVolumeSpecName "kube-api-access-66dsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:49.492947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.492922    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66dsq\" (UniqueName: \"kubernetes.io/projected/a5000257-c412-436d-92f6-9b31c531e94b-kube-api-access-66dsq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:49.553807 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.553777    4860 generic.go:356] "Generic (PLEG): container finished" podID="a5000257-c412-436d-92f6-9b31c531e94b" containerID="01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15" exitCode=0
Oct 29 22:49:49.553946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.553823    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"
Oct 29 22:49:49.553946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.553871    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm" event={"ID":"a5000257-c412-436d-92f6-9b31c531e94b","Type":"ContainerDied","Data":"01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15"}
Oct 29 22:49:49.553946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.553933    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm" event={"ID":"a5000257-c412-436d-92f6-9b31c531e94b","Type":"ContainerDied","Data":"8fe64901c343299ccafa1a7c78d38695a134121ec8a68e0dd23c0566a03802a7"}
Oct 29 22:49:49.554127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.553965    4860 scope.go:122] "RemoveContainer" containerID="01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15"
Oct 29 22:49:49.566501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.566481    4860 scope.go:122] "RemoveContainer" containerID="01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15"
Oct 29 22:49:49.566752 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:49.566731    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15\": container with ID starting with 01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15 not found: ID does not exist" containerID="01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15"
Oct 29 22:49:49.566864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.566761    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15"} err="failed to get container status \"01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15\": rpc error: code = NotFound desc = could not find container \"01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15\": container with ID starting with 01f5bd66b2dd1ab04892bc23776e721549c310ec105f227777e7269043519a15 not found: ID does not exist"
Oct 29 22:49:49.618508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.618427    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"]
Oct 29 22:49:49.630309 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:49.630278    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e619b6-868f869f9f-j8pwm"]
Oct 29 22:49:50.073277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:50.073236    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:50.128366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:50.128294    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5000257-c412-436d-92f6-9b31c531e94b" path="/var/lib/kubelet/pods/a5000257-c412-436d-92f6-9b31c531e94b/volumes"
Oct 29 22:49:50.886483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:50.886419    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv" podUID="f33354fa-3db1-4e3c-8cf3-65ff12a69ef4" containerName="payload-pre-processing" containerID="cri-o://2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682" gracePeriod=25
Oct 29 22:49:51.199216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:51.199181    4860 patch_prober.go:29] interesting pod/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:49:51.199407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:51.199255    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" podUID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:49:51.663588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:51.663510    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:51.985088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:51.985053    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:53.147915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.147884    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:53.377436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.377173    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv" podUID="f33354fa-3db1-4e3c-8cf3-65ff12a69ef4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:49:53.528217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.527833    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:49:53.619348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.618200    4860 generic.go:356] "Generic (PLEG): container finished" podID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" containerID="12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a" exitCode=0
Oct 29 22:49:53.619348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.618519    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"
Oct 29 22:49:53.619348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.618536    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" event={"ID":"902c65e5-dea2-4afc-93c0-8b5b1b46bae6","Type":"ContainerDied","Data":"12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a"}
Oct 29 22:49:53.619348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.618589    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h" event={"ID":"902c65e5-dea2-4afc-93c0-8b5b1b46bae6","Type":"ContainerDied","Data":"09f8b3da6933abcdd2425e5b662c49613552e6d82c73c1615d583a1b66edbe8a"}
Oct 29 22:49:53.619348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.618620    4860 scope.go:122] "RemoveContainer" containerID="12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a"
Oct 29 22:49:53.637801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.637780    4860 scope.go:122] "RemoveContainer" containerID="12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a"
Oct 29 22:49:53.638176 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:53.638140    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a\": container with ID starting with 12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a not found: ID does not exist" containerID="12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a"
Oct 29 22:49:53.638405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.638379    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a"} err="failed to get container status \"12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a\": rpc error: code = NotFound desc = could not find container \"12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a\": container with ID starting with 12f7e0138ef974a30dcc284ae410f65943c34d88d88afd6f4085e73b87e6e59a not found: ID does not exist"
Oct 29 22:49:53.680126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.680084    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-certs\") pod \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") "
Oct 29 22:49:53.680345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.680269    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-envoy\") pod \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") "
Oct 29 22:49:53.680426 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.680402    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-kube-api-access-rbg4s\" (UniqueName: \"kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-kube-api-access-rbg4s\") pod \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") "
Oct 29 22:49:53.680490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.680413    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-certs" pod "902c65e5-dea2-4afc-93c0-8b5b1b46bae6" (UID: "902c65e5-dea2-4afc-93c0-8b5b1b46bae6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:53.680550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.680483    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-podinfo\") pod \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") "
Oct 29 22:49:53.680550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.680531    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-credential-socket\") pod \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") "
Oct 29 22:49:53.680670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.680583    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-socket\") pod \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") "
Oct 29 22:49:53.680670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.680625    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-data\") pod \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") "
Oct 29 22:49:53.680784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.680669    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-token\" (UniqueName: \"kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-token\") pod \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") "
Oct 29 22:49:53.680784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.680742    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istiod-ca-cert\") pod \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\" (UID: \"902c65e5-dea2-4afc-93c0-8b5b1b46bae6\") "
Oct 29 22:49:53.681786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.681303    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-socket" pod "902c65e5-dea2-4afc-93c0-8b5b1b46bae6" (UID: "902c65e5-dea2-4afc-93c0-8b5b1b46bae6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:53.681786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.681343    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.681786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.681546    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-credential-socket" pod "902c65e5-dea2-4afc-93c0-8b5b1b46bae6" (UID: "902c65e5-dea2-4afc-93c0-8b5b1b46bae6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:53.681786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.681646    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istiod-ca-cert" pod "902c65e5-dea2-4afc-93c0-8b5b1b46bae6" (UID: "902c65e5-dea2-4afc-93c0-8b5b1b46bae6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:53.682096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.681822    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:53.682096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.681987    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-data" pod "902c65e5-dea2-4afc-93c0-8b5b1b46bae6" (UID: "902c65e5-dea2-4afc-93c0-8b5b1b46bae6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:53.683370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.683340    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-kube-api-access-rbg4s" pod "902c65e5-dea2-4afc-93c0-8b5b1b46bae6" (UID: "902c65e5-dea2-4afc-93c0-8b5b1b46bae6"). InnerVolumeSpecName "kube-api-access-rbg4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:53.683497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.683384    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-envoy" pod "902c65e5-dea2-4afc-93c0-8b5b1b46bae6" (UID: "902c65e5-dea2-4afc-93c0-8b5b1b46bae6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:53.684630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.684603    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-token" pod "902c65e5-dea2-4afc-93c0-8b5b1b46bae6" (UID: "902c65e5-dea2-4afc-93c0-8b5b1b46bae6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:53.684860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.684831    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-podinfo" pod "902c65e5-dea2-4afc-93c0-8b5b1b46bae6" (UID: "902c65e5-dea2-4afc-93c0-8b5b1b46bae6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:53.782670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.782637    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.782670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.782668    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbg4s\" (UniqueName: \"kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-kube-api-access-rbg4s\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.782670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.782679    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.782910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.782688    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.782910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.782697    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.782910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.782707    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.782910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.782716    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.782910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.782724    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902c65e5-dea2-4afc-93c0-8b5b1b46bae6-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:53.820535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.820498    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:53.969463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.969102    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:53.989373 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.980139    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"]
Oct 29 22:49:53.997829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:53.997742    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8147a8-openshift-default-855c99786c-4sv5h"]
Oct 29 22:49:54.002696 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:54.002668    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"]
Oct 29 22:49:54.002971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:54.002944    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy" containerID="cri-o://b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce" gracePeriod=30
Oct 29 22:49:54.110340 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:54.110285    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:54.121665 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:54.121586    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" path="/var/lib/kubelet/pods/902c65e5-dea2-4afc-93c0-8b5b1b46bae6/volumes"
Oct 29 22:49:54.219316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:54.219277    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"]
Oct 29 22:49:54.226694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:54.226613    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2a3ef0-j4d57"]
Oct 29 22:49:54.232412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:54.232380    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:54.704861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:54.704822    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:54.903170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:54.903137    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:55.380304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:55.380268    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:55.512564 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:55.512199    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:56.113299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:56.113269    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ffe539a-ae7b-4878-b2fd-7c389014bbe0" path="/var/lib/kubelet/pods/2ffe539a-ae7b-4878-b2fd-7c389014bbe0/volumes"
Oct 29 22:49:56.115677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:56.115648    4860 patch_prober.go:29] interesting pod/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:49:56.115839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:56.115707    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:49:56.345337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:56.345298    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:58.069705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:58.069675    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:58.861618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:58.861585    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:58.999883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:58.999850    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:59.292169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.292145    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:49:59.368268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.368233    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-token\" (UniqueName: \"kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-token\") pod \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") "
Oct 29 22:49:59.368442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.368345    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istiod-ca-cert\") pod \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") "
Oct 29 22:49:59.368442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.368420    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-data\") pod \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") "
Oct 29 22:49:59.368573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.368465    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-kube-api-access-q5829\" (UniqueName: \"kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-kube-api-access-q5829\") pod \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") "
Oct 29 22:49:59.368573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.368563    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-socket\") pod \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") "
Oct 29 22:49:59.368683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.368601    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-certs\") pod \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") "
Oct 29 22:49:59.368683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.368639    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-podinfo\") pod \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") "
Oct 29 22:49:59.368806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.368700    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-credential-socket\") pod \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") "
Oct 29 22:49:59.368806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.368731    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-envoy\") pod \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\" (UID: \"a9748a13-8e36-4b49-bc3d-ac909ea8d367\") "
Oct 29 22:49:59.369892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.369144    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-certs" pod "a9748a13-8e36-4b49-bc3d-ac909ea8d367" (UID: "a9748a13-8e36-4b49-bc3d-ac909ea8d367"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.369892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.369431    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.369892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.369637    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-credential-socket" pod "a9748a13-8e36-4b49-bc3d-ac909ea8d367" (UID: "a9748a13-8e36-4b49-bc3d-ac909ea8d367"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.369892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.369653    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-socket" pod "a9748a13-8e36-4b49-bc3d-ac909ea8d367" (UID: "a9748a13-8e36-4b49-bc3d-ac909ea8d367"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.369892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.369721    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-data" pod "a9748a13-8e36-4b49-bc3d-ac909ea8d367" (UID: "a9748a13-8e36-4b49-bc3d-ac909ea8d367"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.369892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.369799    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istiod-ca-cert" pod "a9748a13-8e36-4b49-bc3d-ac909ea8d367" (UID: "a9748a13-8e36-4b49-bc3d-ac909ea8d367"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:49:59.375386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.375287    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-token" pod "a9748a13-8e36-4b49-bc3d-ac909ea8d367" (UID: "a9748a13-8e36-4b49-bc3d-ac909ea8d367"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:59.376385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.376308    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-podinfo" pod "a9748a13-8e36-4b49-bc3d-ac909ea8d367" (UID: "a9748a13-8e36-4b49-bc3d-ac909ea8d367"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:49:59.376495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.376403    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-kube-api-access-q5829" pod "a9748a13-8e36-4b49-bc3d-ac909ea8d367" (UID: "a9748a13-8e36-4b49-bc3d-ac909ea8d367"). InnerVolumeSpecName "kube-api-access-q5829". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:49:59.384648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.384609    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-envoy" pod "a9748a13-8e36-4b49-bc3d-ac909ea8d367" (UID: "a9748a13-8e36-4b49-bc3d-ac909ea8d367"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:49:59.471041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.471007    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.471185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.471045    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.471185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.471062    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.471185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.471075    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.471185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.471088    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.471185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.471101    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5829\" (UniqueName: \"kubernetes.io/projected/a9748a13-8e36-4b49-bc3d-ac909ea8d367-kube-api-access-q5829\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.471185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.471115    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9748a13-8e36-4b49-bc3d-ac909ea8d367-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.471185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.471127    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9748a13-8e36-4b49-bc3d-ac909ea8d367-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:49:59.686530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.686499    4860 generic.go:356] "Generic (PLEG): container finished" podID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerID="b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce" exitCode=0
Oct 29 22:49:59.686719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.686548    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"
Oct 29 22:49:59.686719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.686590    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" event={"ID":"a9748a13-8e36-4b49-bc3d-ac909ea8d367","Type":"ContainerDied","Data":"b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce"}
Oct 29 22:49:59.686719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.686648    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp" event={"ID":"a9748a13-8e36-4b49-bc3d-ac909ea8d367","Type":"ContainerDied","Data":"e6de5d18ae6c89c372cea6f1c9570d7fc3af03e023a541f24f512bb3075edbe6"}
Oct 29 22:49:59.686719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.686678    4860 scope.go:122] "RemoveContainer" containerID="b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce"
Oct 29 22:49:59.700019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.699988    4860 scope.go:122] "RemoveContainer" containerID="a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68"
Oct 29 22:49:59.710603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.710586    4860 scope.go:122] "RemoveContainer" containerID="b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce"
Oct 29 22:49:59.710926 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:59.710906    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce\": container with ID starting with b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce not found: ID does not exist" containerID="b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce"
Oct 29 22:49:59.711029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.710932    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce"} err="failed to get container status \"b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce\": rpc error: code = NotFound desc = could not find container \"b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce\": container with ID starting with b8bb3f3b7ad325486eb3563f9ffd0e90413e23c39b7a2c8cf14f569cbdec1cce not found: ID does not exist"
Oct 29 22:49:59.711029 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.710949    4860 scope.go:122] "RemoveContainer" containerID="a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68"
Oct 29 22:49:59.711228 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:49:59.711210    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68\": container with ID starting with a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68 not found: ID does not exist" containerID="a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68"
Oct 29 22:49:59.711295 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.711234    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68"} err="failed to get container status \"a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68\": rpc error: code = NotFound desc = could not find container \"a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68\": container with ID starting with a78976df1ed808b763440edc41f41de00f046ca4ccff6f4fa79d7162f9579c68 not found: ID does not exist"
Oct 29 22:49:59.747925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.747904    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"]
Oct 29 22:49:59.761260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.761241    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e619b6-openshift-default-6d4c644df9-kv7vp"]
Oct 29 22:49:59.886588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:49:59.886560    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:00.113193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:00.113119    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" path="/var/lib/kubelet/pods/a9748a13-8e36-4b49-bc3d-ac909ea8d367/volumes"
Oct 29 22:50:00.672142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:00.672112    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:00.812374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:00.812348    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:01.464517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:01.464481    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:01.595268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:01.595236    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:01.750894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:01.750795    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:01.755729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:01.755697    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"]
Oct 29 22:50:01.756026 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:01.755998    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs" podUID="b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" containerName="istio-proxy" containerID="cri-o://6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e" gracePeriod=30
Oct 29 22:50:01.893767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:01.888562    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:02.606811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:02.606771    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:03.128819 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:03.128780    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:03.272191 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:03.272155    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:03.363667 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:03.363623    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv" podUID="f33354fa-3db1-4e3c-8cf3-65ff12a69ef4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:50:03.733143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:03.733107    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:04.794481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.794443    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"]
Oct 29 22:50:04.795427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795405    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy"
Oct 29 22:50:04.795519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795429    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy"
Oct 29 22:50:04.795519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795440    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9014903f-7eee-4a29-bec9-933b6f83ca47" containerName="revoke-keys"
Oct 29 22:50:04.795519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795446    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9014903f-7eee-4a29-bec9-933b6f83ca47" containerName="revoke-keys"
Oct 29 22:50:04.795519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795465    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a5000257-c412-436d-92f6-9b31c531e94b" containerName="payload-pre-processing"
Oct 29 22:50:04.795519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795470    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5000257-c412-436d-92f6-9b31c531e94b" containerName="payload-pre-processing"
Oct 29 22:50:04.795519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795499    4860 cpu_manager.go:414] "containerMap: removing container" podUID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" containerName="istio-proxy"
Oct 29 22:50:04.795519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795505    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" containerName="istio-proxy"
Oct 29 22:50:04.795779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795532    4860 cpu_manager.go:414] "containerMap: removing container" podUID="087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" containerName="istio-proxy"
Oct 29 22:50:04.795779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795537    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" containerName="istio-proxy"
Oct 29 22:50:04.795779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795654    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="087e8af1-5dcb-4f4a-8a0e-7ab3339eeb09" containerName="istio-proxy"
Oct 29 22:50:04.795779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795667    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy"
Oct 29 22:50:04.795779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795675    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9014903f-7eee-4a29-bec9-933b6f83ca47" containerName="revoke-keys"
Oct 29 22:50:04.795779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795681    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy"
Oct 29 22:50:04.795779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795689    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5000257-c412-436d-92f6-9b31c531e94b" containerName="payload-pre-processing"
Oct 29 22:50:04.795779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.795697    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="902c65e5-dea2-4afc-93c0-8b5b1b46bae6" containerName="istio-proxy"
Oct 29 22:50:04.801708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.801685    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:04.823197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.823168    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"]
Oct 29 22:50:04.830541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.830516    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-c933cb-serving-cert\""
Oct 29 22:50:04.866281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.866252    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:04.866429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.866301    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfxf8\" (UniqueName: \"kubernetes.io/projected/1d25d5c6-0825-4c7e-9926-4ab0add384c4-kube-api-access-tfxf8\") pod \"maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:04.866429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.866360    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:04.866607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.866585    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d25d5c6-0825-4c7e-9926-4ab0add384c4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:04.912270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.912240    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:04.967342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.967301    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:04.967462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.967392    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfxf8\" (UniqueName: \"kubernetes.io/projected/1d25d5c6-0825-4c7e-9926-4ab0add384c4-kube-api-access-tfxf8\") pod \"maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:04.967462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.967432    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:04.967554 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.967538    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d25d5c6-0825-4c7e-9926-4ab0add384c4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:04.968146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.968125    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d25d5c6-0825-4c7e-9926-4ab0add384c4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:04.969920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.969897    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:04.970004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.969920    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:04.988839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:04.988812    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfxf8\" (UniqueName: \"kubernetes.io/projected/1d25d5c6-0825-4c7e-9926-4ab0add384c4-kube-api-access-tfxf8\") pod \"maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:05.033106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:05.033073    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:05.117926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:05.117847    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:05.153115 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:05.153036    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:05.434725 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:50:05.434696    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d25d5c6_0825_4c7e_9926_4ab0add384c4.slice/crio-c71eea82a796446f84f8d377a3326c40bac253f9e72f547049b3a3dac2af2d90 WatchSource:0}: Error finding container c71eea82a796446f84f8d377a3326c40bac253f9e72f547049b3a3dac2af2d90: Status 404 returned error can't find the container with id c71eea82a796446f84f8d377a3326c40bac253f9e72f547049b3a3dac2af2d90
Oct 29 22:50:05.437219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:05.437197    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"]
Oct 29 22:50:05.778441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:05.778358    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb" event={"ID":"1d25d5c6-0825-4c7e-9926-4ab0add384c4","Type":"ContainerStarted","Data":"c71eea82a796446f84f8d377a3326c40bac253f9e72f547049b3a3dac2af2d90"}
Oct 29 22:50:06.400954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:06.400909    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:06.792123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:06.792094    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb" event={"ID":"1d25d5c6-0825-4c7e-9926-4ab0add384c4","Type":"ContainerStarted","Data":"35536d3ab530f49d5ede8fcb1342d2b77822c52630a68124566af062d55f066d"}
Oct 29 22:50:06.792231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:06.792193    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:06.838006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:06.837947    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb" podStartSLOduration=2.422683444 podStartE2EDuration="2.837928913s" totalImagesPullingTime="415.245469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:06.835773409 +0000 UTC m=+2148.860549563" watchObservedRunningTime="2026-10-29 22:50:06.837928913 +0000 UTC m=+2148.862705045"
Oct 29 22:50:06.972943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:06.972910    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:07.077285 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.077254    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"]
Oct 29 22:50:07.081738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.081712    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-05417d-sxgxf"]
Oct 29 22:50:07.101896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.101515    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:50:07.115581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.115551    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:07.208810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.208780    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-token\" (UniqueName: \"kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-token\") pod \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") "
Oct 29 22:50:07.208961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.208833    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-kube-api-access-9d6kc\" (UniqueName: \"kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-kube-api-access-9d6kc\") pod \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") "
Oct 29 22:50:07.208961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.208919    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-certs\") pod \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") "
Oct 29 22:50:07.209079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.208975    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-envoy\") pod \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") "
Oct 29 22:50:07.209079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.209001    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-credential-socket\") pod \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") "
Oct 29 22:50:07.209079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.209040    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istiod-ca-cert\") pod \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") "
Oct 29 22:50:07.209254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.209118    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-data\") pod \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") "
Oct 29 22:50:07.209254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.209148    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-socket\") pod \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") "
Oct 29 22:50:07.209254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.209221    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-podinfo\") pod \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\" (UID: \"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1\") "
Oct 29 22:50:07.209479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.209245    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-certs" pod "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" (UID: "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:07.209479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.209371    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-credential-socket" pod "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" (UID: "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:07.209591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.209539    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-socket" pod "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" (UID: "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:07.209653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.209617    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istiod-ca-cert" pod "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" (UID: "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:50:07.210403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.210380    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.210537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.210407    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.210537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.210423    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.210537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.210437    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.210537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.210519    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-data" pod "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" (UID: "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:07.211494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.211468    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-envoy" pod "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" (UID: "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:07.211580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.211518    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-kube-api-access-9d6kc" pod "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" (UID: "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1"). InnerVolumeSpecName "kube-api-access-9d6kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:07.211621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.211589    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-token" pod "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" (UID: "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:07.211681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.211661    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-podinfo" pod "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" (UID: "b3538b7e-90e3-4cbc-8d32-a82d02f0dae1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:50:07.311387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.311360    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.311387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.311389    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.311558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.311398    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d6kc\" (UniqueName: \"kubernetes.io/projected/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-kube-api-access-9d6kc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.311558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.311408    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.311558 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.311417    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:07.316054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.315995    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:07.459703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.459666    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:07.614200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.614129    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:07.804559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.804499    4860 generic.go:356] "Generic (PLEG): container finished" podID="b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" containerID="6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e" exitCode=0
Oct 29 22:50:07.804738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.804608    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"
Oct 29 22:50:07.804738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.804645    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs" event={"ID":"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1","Type":"ContainerDied","Data":"6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e"}
Oct 29 22:50:07.804738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.804710    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs" event={"ID":"b3538b7e-90e3-4cbc-8d32-a82d02f0dae1","Type":"ContainerDied","Data":"98e6f7e2e7b3773c1247e814353ec57ec8d485eb6f384cdfd44bf77ddc32f463"}
Oct 29 22:50:07.804884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.804744    4860 scope.go:122] "RemoveContainer" containerID="6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e"
Oct 29 22:50:07.819468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.819445    4860 scope.go:122] "RemoveContainer" containerID="6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e"
Oct 29 22:50:07.819739 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:50:07.819721    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e\": container with ID starting with 6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e not found: ID does not exist" containerID="6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e"
Oct 29 22:50:07.819827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.819747    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e"} err="failed to get container status \"6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e\": rpc error: code = NotFound desc = could not find container \"6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e\": container with ID starting with 6eb6be6c4577960cc25685fff54146d45fe93f557c061decec03fd76e235836e not found: ID does not exist"
Oct 29 22:50:07.885097 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.885017    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"]
Oct 29 22:50:07.898082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:07.898041    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2a3ef0-openshift-default-5cdf89c9-rvbgs"]
Oct 29 22:50:08.122166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:08.122124    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9014903f-7eee-4a29-bec9-933b6f83ca47" path="/var/lib/kubelet/pods/9014903f-7eee-4a29-bec9-933b6f83ca47/volumes"
Oct 29 22:50:08.123044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:08.123012    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" path="/var/lib/kubelet/pods/b3538b7e-90e3-4cbc-8d32-a82d02f0dae1/volumes"
Oct 29 22:50:09.198350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:09.194093    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:09.442623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:09.442578    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:11.083442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.083045    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:11.253185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.253161    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"
Oct 29 22:50:11.434642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.434613    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-kube-api-access-gvtwr\" (UniqueName: \"kubernetes.io/projected/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-kube-api-access-gvtwr\") pod \"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4\" (UID: \"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4\") "
Oct 29 22:50:11.434830 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.434664    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-plugins-config-volume\") pod \"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4\" (UID: \"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4\") "
Oct 29 22:50:11.435127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.435103    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-plugins-config-volume" pod "f33354fa-3db1-4e3c-8cf3-65ff12a69ef4" (UID: "f33354fa-3db1-4e3c-8cf3-65ff12a69ef4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:50:11.435238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.435223    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:11.436767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.436748    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-kube-api-access-gvtwr" pod "f33354fa-3db1-4e3c-8cf3-65ff12a69ef4" (UID: "f33354fa-3db1-4e3c-8cf3-65ff12a69ef4"). InnerVolumeSpecName "kube-api-access-gvtwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:11.536132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.536062    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvtwr\" (UniqueName: \"kubernetes.io/projected/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4-kube-api-access-gvtwr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:11.959158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.959119    4860 generic.go:356] "Generic (PLEG): container finished" podID="f33354fa-3db1-4e3c-8cf3-65ff12a69ef4" containerID="2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682" exitCode=0
Oct 29 22:50:11.959401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.959169    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"
Oct 29 22:50:11.959401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.959215    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv" event={"ID":"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4","Type":"ContainerDied","Data":"2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682"}
Oct 29 22:50:11.959401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.959277    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv" event={"ID":"f33354fa-3db1-4e3c-8cf3-65ff12a69ef4","Type":"ContainerDied","Data":"d2a101e1f306008d29c46a78cf892113bad0995aad22604d1cb6a3091e3e4a1a"}
Oct 29 22:50:11.959401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.959309    4860 scope.go:122] "RemoveContainer" containerID="2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682"
Oct 29 22:50:11.998213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.998189    4860 scope.go:122] "RemoveContainer" containerID="2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682"
Oct 29 22:50:11.998567 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:50:11.998536    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682\": container with ID starting with 2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682 not found: ID does not exist" containerID="2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682"
Oct 29 22:50:11.998670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:11.998565    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682"} err="failed to get container status \"2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682\": rpc error: code = NotFound desc = could not find container \"2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682\": container with ID starting with 2430ed794dece748e1474459819d647b4479a4900738d99665e78b8274eae682 not found: ID does not exist"
Oct 29 22:50:12.042487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:12.042457    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"]
Oct 29 22:50:12.055305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:12.055271    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-05417d-5cbcfc559-jjksv"]
Oct 29 22:50:12.112735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:12.112705    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33354fa-3db1-4e3c-8cf3-65ff12a69ef4" path="/var/lib/kubelet/pods/f33354fa-3db1-4e3c-8cf3-65ff12a69ef4/volumes"
Oct 29 22:50:12.811536 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:12.811505    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 22:50:12.862920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:12.862882    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:13.593258 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:13.593224    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:14.009703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:14.009666    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:14.131624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:14.131587    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:14.306186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:14.306091    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"]
Oct 29 22:50:14.306395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:14.306358    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerName="istio-proxy" containerID="cri-o://b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49" gracePeriod=30
Oct 29 22:50:14.448349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:14.447822    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:14.634532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:14.634451    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:16.471053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:16.471011    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:17.194473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:17.194433    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:50:17.194724 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:17.194480    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:50:18.058647 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:50:18.058612    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:50:18.157649 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:18.157618    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:18.724834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:18.724804    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:19.064787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:19.064709    4860 generic.go:356] "Generic (PLEG): container finished" podID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerID="e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70" exitCode=0
Oct 29 22:50:19.065195 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:19.064782    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" event={"ID":"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0","Type":"ContainerDied","Data":"e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70"}
Oct 29 22:50:19.065272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:19.065215    4860 scope.go:122] "RemoveContainer" containerID="e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70"
Oct 29 22:50:19.066951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:19.066921    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:50:19.067023 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:19.066991    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:50:19.067073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:19.067037    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:50:19.885478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:19.885440    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:20.010069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.010036    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:50:20.093921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.088371    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" event={"ID":"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0","Type":"ContainerStarted","Data":"cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2"}
Oct 29 22:50:20.093921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.092562    4860 generic.go:356] "Generic (PLEG): container finished" podID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerID="b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49" exitCode=0
Oct 29 22:50:20.093921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.092854    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" event={"ID":"c897948d-a9ba-4e67-81c5-d0830ac823cd","Type":"ContainerDied","Data":"b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49"}
Oct 29 22:50:20.093921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.092932    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg" event={"ID":"c897948d-a9ba-4e67-81c5-d0830ac823cd","Type":"ContainerDied","Data":"d8991f6e437b07fb3680eda82579c04f55930ec5b3c22c7617986b3981381d15"}
Oct 29 22:50:20.093921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.092971    4860 scope.go:122] "RemoveContainer" containerID="b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49"
Oct 29 22:50:20.093921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.093538    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"
Oct 29 22:50:20.099898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.099865    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-data\") pod \"c897948d-a9ba-4e67-81c5-d0830ac823cd\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") "
Oct 29 22:50:20.100044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.100005    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-token\" (UniqueName: \"kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-token\") pod \"c897948d-a9ba-4e67-81c5-d0830ac823cd\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") "
Oct 29 22:50:20.100119 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.100101    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-kube-api-access-qkxrq\" (UniqueName: \"kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-kube-api-access-qkxrq\") pod \"c897948d-a9ba-4e67-81c5-d0830ac823cd\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") "
Oct 29 22:50:20.100189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.100169    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-podinfo\") pod \"c897948d-a9ba-4e67-81c5-d0830ac823cd\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") "
Oct 29 22:50:20.100251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.100209    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-envoy\") pod \"c897948d-a9ba-4e67-81c5-d0830ac823cd\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") "
Oct 29 22:50:20.100308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.100270    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-certs\") pod \"c897948d-a9ba-4e67-81c5-d0830ac823cd\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") "
Oct 29 22:50:20.100410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.100395    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-credential-socket\") pod \"c897948d-a9ba-4e67-81c5-d0830ac823cd\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") "
Oct 29 22:50:20.100485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.100450    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-socket\") pod \"c897948d-a9ba-4e67-81c5-d0830ac823cd\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") "
Oct 29 22:50:20.100545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.100530    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c897948d-a9ba-4e67-81c5-d0830ac823cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c897948d-a9ba-4e67-81c5-d0830ac823cd-istiod-ca-cert\") pod \"c897948d-a9ba-4e67-81c5-d0830ac823cd\" (UID: \"c897948d-a9ba-4e67-81c5-d0830ac823cd\") "
Oct 29 22:50:20.100604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.100530    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-data" pod "c897948d-a9ba-4e67-81c5-d0830ac823cd" (UID: "c897948d-a9ba-4e67-81c5-d0830ac823cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:20.101452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.101426    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:20.101707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.101684    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-certs" pod "c897948d-a9ba-4e67-81c5-d0830ac823cd" (UID: "c897948d-a9ba-4e67-81c5-d0830ac823cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:20.101917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.101884    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-credential-socket" pod "c897948d-a9ba-4e67-81c5-d0830ac823cd" (UID: "c897948d-a9ba-4e67-81c5-d0830ac823cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:20.102113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.102090    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-socket" pod "c897948d-a9ba-4e67-81c5-d0830ac823cd" (UID: "c897948d-a9ba-4e67-81c5-d0830ac823cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:20.102444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.102421    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c897948d-a9ba-4e67-81c5-d0830ac823cd-istiod-ca-cert" pod "c897948d-a9ba-4e67-81c5-d0830ac823cd" (UID: "c897948d-a9ba-4e67-81c5-d0830ac823cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:50:20.138396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.138317    4860 scope.go:122] "RemoveContainer" containerID="e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f"
Oct 29 22:50:20.141889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.141854    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-token" pod "c897948d-a9ba-4e67-81c5-d0830ac823cd" (UID: "c897948d-a9ba-4e67-81c5-d0830ac823cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:20.142825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.142795    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-envoy" pod "c897948d-a9ba-4e67-81c5-d0830ac823cd" (UID: "c897948d-a9ba-4e67-81c5-d0830ac823cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:50:20.143813 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.143787    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-kube-api-access-qkxrq" pod "c897948d-a9ba-4e67-81c5-d0830ac823cd" (UID: "c897948d-a9ba-4e67-81c5-d0830ac823cd"). InnerVolumeSpecName "kube-api-access-qkxrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:50:20.145511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.145489    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-podinfo" pod "c897948d-a9ba-4e67-81c5-d0830ac823cd" (UID: "c897948d-a9ba-4e67-81c5-d0830ac823cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:50:20.196596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.196535    4860 scope.go:122] "RemoveContainer" containerID="b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49"
Oct 29 22:50:20.196995 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:50:20.196965    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49\": container with ID starting with b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49 not found: ID does not exist" containerID="b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49"
Oct 29 22:50:20.197092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.197008    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49"} err="failed to get container status \"b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49\": rpc error: code = NotFound desc = could not find container \"b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49\": container with ID starting with b2ecaf4a8a0a544c72381981b86b4cda532121d76de3e6a5f6ef0913f2690a49 not found: ID does not exist"
Oct 29 22:50:20.197092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.197033    4860 scope.go:122] "RemoveContainer" containerID="e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f"
Oct 29 22:50:20.197393 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:50:20.197364    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f\": container with ID starting with e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f not found: ID does not exist" containerID="e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f"
Oct 29 22:50:20.197491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.197401    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f"} err="failed to get container status \"e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f\": rpc error: code = NotFound desc = could not find container \"e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f\": container with ID starting with e67f384d80b82521dc105c56dbdd0bd7006575acbf3d1a42399a1b9d9f49800f not found: ID does not exist"
Oct 29 22:50:20.202641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.202616    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:20.202641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.202641    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkxrq\" (UniqueName: \"kubernetes.io/projected/c897948d-a9ba-4e67-81c5-d0830ac823cd-kube-api-access-qkxrq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:20.202795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.202656    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:20.202795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.202668    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:20.202795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.202680    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:20.202795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.202692    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:20.202795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.202704    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c897948d-a9ba-4e67-81c5-d0830ac823cd-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:20.202795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.202717    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c897948d-a9ba-4e67-81c5-d0830ac823cd-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:50:20.442083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.442041    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"]
Oct 29 22:50:20.446343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.446303    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-05417d-openshift-default-6d96dbcbc5-nq8jg"]
Oct 29 22:50:20.519557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.519527    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:50:20.519736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.519686    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 29 22:50:20.519794 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:20.519728    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 29 22:50:21.519517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:21.519478    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 29 22:50:21.519940 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:21.519535    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 29 22:50:21.749020 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:21.748981    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:22.113616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:22.113585    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" path="/var/lib/kubelet/pods/c897948d-a9ba-4e67-81c5-d0830ac823cd/volumes"
Oct 29 22:50:22.519947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:22.519915    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 29 22:50:22.520379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:22.519964    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 29 22:50:23.108243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:23.108210    4860 scope.go:122] "RemoveContainer" containerID="f3805db84d8c0ebfabf301a304c3b4c30b776720c41e652197590400594507c5"
Oct 29 22:50:23.119444 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:23.119422    4860 scope.go:122] "RemoveContainer" containerID="091f5ceb9a43ac10c55c602762ed98133aa7aef3f9b755820d810f5898ceb307"
Oct 29 22:50:23.397224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:23.397100    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:23.519565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:23.519530    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 29 22:50:23.519747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:23.519587    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 29 22:50:23.793197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:23.793134    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:23.918417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:23.918374    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:24.519438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:24.519394    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 29 22:50:24.519653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:24.519481    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 29 22:50:24.650725 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:24.650694    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:25.284349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:25.281920    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:25.519929 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:25.519893    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 29 22:50:25.520113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:25.519955    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 29 22:50:26.523806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:26.523775    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:50:26.524315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:26.524282    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:50:26.525073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:26.525052    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 22:50:27.177434 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:27.177402    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:28.902048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:28.902013    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:30.444516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.444479    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"]
Oct 29 22:50:30.445181 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445162    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f33354fa-3db1-4e3c-8cf3-65ff12a69ef4" containerName="payload-pre-processing"
Oct 29 22:50:30.445243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445183    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33354fa-3db1-4e3c-8cf3-65ff12a69ef4" containerName="payload-pre-processing"
Oct 29 22:50:30.445243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445202    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" containerName="istio-proxy"
Oct 29 22:50:30.445243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445207    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" containerName="istio-proxy"
Oct 29 22:50:30.445243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445236    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerName="istio-proxy"
Oct 29 22:50:30.445243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445241    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerName="istio-proxy"
Oct 29 22:50:30.445446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445254    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerName="istio-proxy"
Oct 29 22:50:30.445446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445259    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerName="istio-proxy"
Oct 29 22:50:30.445446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445272    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy"
Oct 29 22:50:30.445446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445277    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9748a13-8e36-4b49-bc3d-ac909ea8d367" containerName="istio-proxy"
Oct 29 22:50:30.445446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445404    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerName="istio-proxy"
Oct 29 22:50:30.445446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445417    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33354fa-3db1-4e3c-8cf3-65ff12a69ef4" containerName="payload-pre-processing"
Oct 29 22:50:30.445446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.445425    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3538b7e-90e3-4cbc-8d32-a82d02f0dae1" containerName="istio-proxy"
Oct 29 22:50:30.450831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.450809    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.469735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.469706    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"]
Oct 29 22:50:30.479174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.479143    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-920032-serving-cert\""
Oct 29 22:50:30.557875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.557812    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:30.563704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.563673    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.563825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.563758    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c970b9-cd72-4a45-a8de-317f6d556a5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.563899 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.563873    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.564028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.564009    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zpc4\" (UniqueName: \"kubernetes.io/projected/00c970b9-cd72-4a45-a8de-317f6d556a5a-kube-api-access-5zpc4\") pod \"maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.681777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.678273    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.681777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.678355    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c970b9-cd72-4a45-a8de-317f6d556a5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.681777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.678521    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.681777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.678637    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zpc4\" (UniqueName: \"kubernetes.io/projected/00c970b9-cd72-4a45-a8de-317f6d556a5a-kube-api-access-5zpc4\") pod \"maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.688500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.688120    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c970b9-cd72-4a45-a8de-317f6d556a5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.700354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.700257    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.710535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.710496    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:30.712622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.712593    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.717000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.716971    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zpc4\" (UniqueName: \"kubernetes.io/projected/00c970b9-cd72-4a45-a8de-317f6d556a5a-kube-api-access-5zpc4\") pod \"maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:30.767297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:30.767271    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:31.082054 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:50:31.082026    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c970b9_cd72_4a45_a8de_317f6d556a5a.slice/crio-dff7193d401735cc2cb1acf02506fea2a58116863999a0646be39c07ab7afb29 WatchSource:0}: Error finding container dff7193d401735cc2cb1acf02506fea2a58116863999a0646be39c07ab7afb29: Status 404 returned error can't find the container with id dff7193d401735cc2cb1acf02506fea2a58116863999a0646be39c07ab7afb29
Oct 29 22:50:31.084527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:31.084500    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"]
Oct 29 22:50:31.217406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:31.217365    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj" event={"ID":"00c970b9-cd72-4a45-a8de-317f6d556a5a","Type":"ContainerStarted","Data":"dff7193d401735cc2cb1acf02506fea2a58116863999a0646be39c07ab7afb29"}
Oct 29 22:50:32.227581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.227543    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj" event={"ID":"00c970b9-cd72-4a45-a8de-317f6d556a5a","Type":"ContainerStarted","Data":"bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045"}
Oct 29 22:50:32.227989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.227763    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:32.274931 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.274876    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj" podStartSLOduration=1.9937662409999999 podStartE2EDuration="2.274856451s" totalImagesPullingTime="281.09021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:32.273515363 +0000 UTC m=+2174.298291516" watchObservedRunningTime="2026-10-29 22:50:32.274856451 +0000 UTC m=+2174.299632606"
Oct 29 22:50:32.454976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.454940    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:32.816189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.816143    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"]
Oct 29 22:50:32.817359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.817313    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c897948d-a9ba-4e67-81c5-d0830ac823cd" containerName="istio-proxy"
Oct 29 22:50:32.821128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.821113    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:32.853136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.853099    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-aaa3c6-openshift-default-dockercfg-rcl6s\""
Oct 29 22:50:32.860757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.860726    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"]
Oct 29 22:50:32.931100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.931070    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-certs\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:32.931362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.931135    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-data\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:32.931362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.931210    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-token\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:32.931362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.931253    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-envoy\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:32.931573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.931393    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-socket\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:32.931573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.931446    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-credential-socket\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:32.931683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.931593    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-podinfo\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:32.931683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.931649    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istiod-ca-cert\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:32.931802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:32.931717    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xnhs\" (UniqueName: \"kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-kube-api-access-5xnhs\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.033215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.033190    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-certs\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.033368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.033231    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-data\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.033368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.033254    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-token\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.033368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.033277    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-envoy\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.033368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.033303    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-socket\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.033368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.033365    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-credential-socket\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.033728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.033439    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-podinfo\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.033728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.033485    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istiod-ca-cert\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.033728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.033536    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xnhs\" (UniqueName: \"kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-kube-api-access-5xnhs\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.033728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.033614    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-certs\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.033728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.033689    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-data\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.034024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.033763    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-socket\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.034116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.034095    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-credential-socket\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.034276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.034255    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istiod-ca-cert\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.035697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.035677    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-envoy\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.036011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.035994    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-podinfo\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.054719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.054683    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-token\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.054914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.054895    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xnhs\" (UniqueName: \"kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-kube-api-access-5xnhs\") pod \"e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") " pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.136010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.135948    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:33.448631 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:50:33.448599    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e501b2_4c2a_4271_a9f5_5435e6ea3a9c.slice/crio-c4b97328150e21ed20daf5ad5196aa0337b6318d9a7646bf9fa3cff380b95f58 WatchSource:0}: Error finding container c4b97328150e21ed20daf5ad5196aa0337b6318d9a7646bf9fa3cff380b95f58: Status 404 returned error can't find the container with id c4b97328150e21ed20daf5ad5196aa0337b6318d9a7646bf9fa3cff380b95f58
Oct 29 22:50:33.451263 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.451229    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:50:33.451353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.451302    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:50:33.451405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.451350    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:50:33.452069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:33.452024    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"]
Oct 29 22:50:34.156369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:34.156316    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:34.259346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:34.259278    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6" event={"ID":"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c","Type":"ContainerStarted","Data":"ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02"}
Oct 29 22:50:34.259545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:34.259382    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6" event={"ID":"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c","Type":"ContainerStarted","Data":"c4b97328150e21ed20daf5ad5196aa0337b6318d9a7646bf9fa3cff380b95f58"}
Oct 29 22:50:35.136736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:35.136702    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:35.138244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:35.138209    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 29 22:50:35.138446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:35.138267    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6" podUID="55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 29 22:50:35.993368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:35.984311    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:36.136259 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:36.136219    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 29 22:50:36.136475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:36.136275    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6" podUID="55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 29 22:50:37.136239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:37.136204    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 29 22:50:37.136641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:37.136260    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6" podUID="55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 29 22:50:37.690350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:37.690187    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:38.136935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:38.136858    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 29 22:50:38.136935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:38.136908    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6" podUID="55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 29 22:50:38.251689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:38.251660    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 22:50:38.329662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:38.329595    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6" podStartSLOduration=6.329575792 podStartE2EDuration="6.329575792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:34.342305793 +0000 UTC m=+2176.367081924" watchObservedRunningTime="2026-10-29 22:50:38.329575792 +0000 UTC m=+2180.354351924"
Oct 29 22:50:39.136357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:39.136293    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 29 22:50:39.136525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:39.136381    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6" podUID="55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 29 22:50:39.366246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:39.366211    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:40.141439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:40.141408    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:40.141895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:40.141869    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:40.142586 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:40.142565    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 22:50:41.218385 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.218355    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:41.342550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.342518    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:41.462416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.462368    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:41.620829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.620753    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"]
Oct 29 22:50:41.626621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.626601    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.649270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.649246    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"]
Oct 29 22:50:41.658133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.658110    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-c933cb-openshift-default-dockercfg-f99fc\""
Oct 29 22:50:41.798712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.798676    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-token\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.798909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.798735    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlsgj\" (UniqueName: \"kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-kube-api-access-wlsgj\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.798909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.798871    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-data\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.799019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.798907    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-podinfo\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.799019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.799000    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.799094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.799074    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-certs\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.799130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.799094    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f864ec3f-6650-4081-9656-cb8f9009b4f0-istiod-ca-cert\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.799253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.799240    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-envoy\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.799311 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.799266    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-credential-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.900839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.900759    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-envoy\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.900839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.900798    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-credential-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.900839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.900830    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-token\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.901139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.900869    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlsgj\" (UniqueName: \"kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-kube-api-access-wlsgj\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.901139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.900914    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-data\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.901139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.900935    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-podinfo\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.901139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.900997    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.901438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.901412    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-certs\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.901508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.901441    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-data\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.901508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.901454    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f864ec3f-6650-4081-9656-cb8f9009b4f0-istiod-ca-cert\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.901508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.901418    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-credential-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.901692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.901658    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.901692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.901679    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-certs\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.901977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.901959    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f864ec3f-6650-4081-9656-cb8f9009b4f0-istiod-ca-cert\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.903280 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.903259    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-envoy\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.903575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.903556    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-podinfo\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.921494 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.921469    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-token\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.922242 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.922222    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlsgj\" (UniqueName: \"kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-kube-api-access-wlsgj\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:41.942006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:41.941987    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:42.259547 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:50:42.259508    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf864ec3f_6650_4081_9656_cb8f9009b4f0.slice/crio-86cd9fae07cd949ba87661f456dd58cbeaaf1c6164cd59e26075ccea78c39959 WatchSource:0}: Error finding container 86cd9fae07cd949ba87661f456dd58cbeaaf1c6164cd59e26075ccea78c39959: Status 404 returned error can't find the container with id 86cd9fae07cd949ba87661f456dd58cbeaaf1c6164cd59e26075ccea78c39959
Oct 29 22:50:42.260453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:42.260431    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"]
Oct 29 22:50:42.262759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:42.262726    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:50:42.262878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:42.262791    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:50:42.262878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:42.262834    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:50:42.388657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:42.388596    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w" event={"ID":"f864ec3f-6650-4081-9656-cb8f9009b4f0","Type":"ContainerStarted","Data":"984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330"}
Oct 29 22:50:42.388657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:42.388658    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w" event={"ID":"f864ec3f-6650-4081-9656-cb8f9009b4f0","Type":"ContainerStarted","Data":"86cd9fae07cd949ba87661f456dd58cbeaaf1c6164cd59e26075ccea78c39959"}
Oct 29 22:50:42.462254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:42.462200    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w" podStartSLOduration=1.462186115 podStartE2EDuration="1.462186115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:42.460391098 +0000 UTC m=+2184.485167229" watchObservedRunningTime="2026-10-29 22:50:42.462186115 +0000 UTC m=+2184.486962245"
Oct 29 22:50:42.670932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:42.670835    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:42.942959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:42.942920    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:43.031087 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:43.031054    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"]
Oct 29 22:50:43.044303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:43.044276    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b17d3f-x5c7p"]
Oct 29 22:50:43.286446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:43.274292    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:43.944751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:43.944715    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 29 22:50:43.944960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:43.944781    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w" podUID="f864ec3f-6650-4081-9656-cb8f9009b4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 29 22:50:44.113881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:44.113852    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64573b13-0260-47f4-87b3-648f478f4f34" path="/var/lib/kubelet/pods/64573b13-0260-47f4-87b3-648f478f4f34/volumes"
Oct 29 22:50:44.942654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:44.942616    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 29 22:50:44.943095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:44.942667    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w" podUID="f864ec3f-6650-4081-9656-cb8f9009b4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 29 22:50:45.330666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:45.330578    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:45.942624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:45.942589    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 29 22:50:45.942798 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:45.942640    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w" podUID="f864ec3f-6650-4081-9656-cb8f9009b4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 29 22:50:46.942930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:46.942896    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 29 22:50:46.943396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:46.942961    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w" podUID="f864ec3f-6650-4081-9656-cb8f9009b4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 29 22:50:47.247117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:47.247032    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:47.942690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:47.942655    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 29 22:50:47.942876 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:47.942719    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w" podUID="f864ec3f-6650-4081-9656-cb8f9009b4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 29 22:50:48.058880 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:50:48.058837    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:50:48.936093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:48.936052    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:48.948089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:48.948063    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:48.948510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:48.948485    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:48.949154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:48.949137    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 22:50:50.797567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:50.797534    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:52.451157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:52.451123    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:54.151919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:54.151884    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:54.277159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:54.277127    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:55.975888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:55.975857    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:57.780167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:57.780136    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:59.477547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:50:59.477516    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:00.034519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:00.034484    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"]
Oct 29 22:51:00.045073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:00.045034    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d31392b-gtxmz"]
Oct 29 22:51:00.113415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:00.113383    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b1b14a-c2db-44fe-9c95-8d0fd0ab90db" path="/var/lib/kubelet/pods/71b1b14a-c2db-44fe-9c95-8d0fd0ab90db/volumes"
Oct 29 22:51:00.978009 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:00.977961    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:01.183984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:01.183943    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:03.062597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:03.062563    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:03.519705 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:03.519674    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:04.787655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:04.787606    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:06.441368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:06.441309    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:08.263427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:08.263390    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:10.026192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:10.026159    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:10.294836 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:10.294758    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:10.760437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:10.760403    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:11.251669 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:11.251630    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:11.378613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:11.378550    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:11.887625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:11.887590    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:13.701824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:13.701784    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:13.821042 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:13.821010    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:15.113756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:15.113724    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:15.248216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:15.248186    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:15.901739 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:15.901710    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:16.806013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:16.805956    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:16.946844 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:16.946807    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:16.969331 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:16.969305    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"]
Oct 29 22:51:16.974043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:16.974019    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"
Oct 29 22:51:16.991150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:16.991126    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"]
Oct 29 22:51:17.000237 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.000214    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-02720a\""
Oct 29 22:51:17.004147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.004121    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw\" (UID: \"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"
Oct 29 22:51:17.004396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.004374    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8wfx\" (UniqueName: \"kubernetes.io/projected/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-kube-api-access-m8wfx\") pod \"payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw\" (UID: \"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"
Oct 29 22:51:17.080503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.066458    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-02720a-dockercfg-b2fr5\""
Oct 29 22:51:17.126318 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.126280    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wfx\" (UniqueName: \"kubernetes.io/projected/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-kube-api-access-m8wfx\") pod \"payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw\" (UID: \"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"
Oct 29 22:51:17.126567 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.126436    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw\" (UID: \"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"
Oct 29 22:51:17.128041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.127982    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw\" (UID: \"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"
Oct 29 22:51:17.149919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.149886    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wfx\" (UniqueName: \"kubernetes.io/projected/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-kube-api-access-m8wfx\") pod \"payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw\" (UID: \"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"
Oct 29 22:51:17.290405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.290369    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"
Oct 29 22:51:17.605937 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:51:17.605904    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5f9159_7bf8_48e0_8fe5_b3cdd3a5a032.slice/crio-f971dc3f505ace94b03287721ba1c16066360d7b7694c88d7900e8e2b3221e38 WatchSource:0}: Error finding container f971dc3f505ace94b03287721ba1c16066360d7b7694c88d7900e8e2b3221e38: Status 404 returned error can't find the container with id f971dc3f505ace94b03287721ba1c16066360d7b7694c88d7900e8e2b3221e38
Oct 29 22:51:17.608603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.608577    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"]
Oct 29 22:51:17.716239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.716210    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw" event={"ID":"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032","Type":"ContainerStarted","Data":"ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d"}
Oct 29 22:51:17.716412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.716277    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"
Oct 29 22:51:17.716412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.716306    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw" event={"ID":"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032","Type":"ContainerStarted","Data":"f971dc3f505ace94b03287721ba1c16066360d7b7694c88d7900e8e2b3221e38"}
Oct 29 22:51:17.759894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.759847    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw" podStartSLOduration=1.75983416 podStartE2EDuration="1.75983416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:17.756706378 +0000 UTC m=+2219.781482519" watchObservedRunningTime="2026-10-29 22:51:17.75983416 +0000 UTC m=+2219.784610291"
Oct 29 22:51:17.904883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:17.904793    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:18.058435 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:51:18.058397    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:51:19.662374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:19.662272    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:21.448429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:21.448391    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:23.232145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:23.232108    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:23.265675 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:23.265645    4860 scope.go:122] "RemoveContainer" containerID="09c59636a48887c35f4bc73aa4c26cd12a8f1c3a20970c4318a7b928b63b6b99"
Oct 29 22:51:23.634998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:23.634925    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:24.074124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.074078    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:24.486202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.486168    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"]
Oct 29 22:51:24.493398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.493371    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.511874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.511846    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-3ee017-openshift-default-dockercfg-xz768\""
Oct 29 22:51:24.547909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.547875    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"]
Oct 29 22:51:24.571131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.571100    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:24.637261 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.637217    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-istio-token\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.655809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.655768    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-envoy\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.669380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.656156    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndc9\" (UniqueName: \"kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-kube-api-access-jndc9\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.669664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.669637    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-credential-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.669789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.669775    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33379dd5-161b-4506-9860-228572c35c00-istiod-ca-cert\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.669907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.669892    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-data\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.670100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.670085    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-certs\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.670239 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.670225    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33379dd5-161b-4506-9860-228572c35c00-istio-podinfo\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.670493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.670476    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.767431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.767296    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:24.772299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.772273    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-istio-token\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.772542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.772520    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-envoy\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.773100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.773077    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndc9\" (UniqueName: \"kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-kube-api-access-jndc9\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.773269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.773250    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-credential-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.773423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.773404    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33379dd5-161b-4506-9860-228572c35c00-istiod-ca-cert\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.773579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.773560    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-data\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.774138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.773713    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-credential-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.774138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.773734    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-certs\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.774138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.773809    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33379dd5-161b-4506-9860-228572c35c00-istio-podinfo\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.774138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.773888    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.774138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.774011    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-certs\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.774778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.774571    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33379dd5-161b-4506-9860-228572c35c00-istiod-ca-cert\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.774778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.774736    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-socket\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.774929 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.774899    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-data\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.777350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.777305    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-envoy\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.780392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.780368    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33379dd5-161b-4506-9860-228572c35c00-istio-podinfo\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.795520 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.795502    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-istio-token\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.799569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.799550    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndc9\" (UniqueName: \"kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-kube-api-access-jndc9\") pod \"e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") " pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.811475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.811454    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:24.895654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:24.895627    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:25.077987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:25.077872    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:25.145372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:25.144065    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:51:25.145372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:25.144185    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:51:25.145372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:25.144227    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:51:25.148869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:25.148840    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"]
Oct 29 22:51:25.792187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:25.792144    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" event={"ID":"33379dd5-161b-4506-9860-228572c35c00","Type":"ContainerStarted","Data":"5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013"}
Oct 29 22:51:25.792625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:25.792213    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" event={"ID":"33379dd5-161b-4506-9860-228572c35c00","Type":"ContainerStarted","Data":"06e09797c5d1691e3b58d010e7b63c841c4424e16d421666b35bcc9bfeacef88"}
Oct 29 22:51:25.812523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:25.812496    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:25.837489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:25.837445    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" podStartSLOduration=1.837432859 podStartE2EDuration="1.837432859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:25.836446223 +0000 UTC m=+2227.861222352" watchObservedRunningTime="2026-10-29 22:51:25.837432859 +0000 UTC m=+2227.862208989"
Oct 29 22:51:26.817755 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:26.817729    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:26.904663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:26.904632    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:27.812133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:27.812097    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:27.813054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:27.813033    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:51:28.298719 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:28.298684    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:28.431721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:28.431682    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:28.727460 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:28.727437    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"
Oct 29 22:51:29.003896 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:29.003803    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:29.997349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:29.994725    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:30.126274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.126243    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:30.150014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.149982    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"]
Oct 29 22:51:30.154735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.154717    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"
Oct 29 22:51:30.173015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.172988    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"]
Oct 29 22:51:30.246421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.246386    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzjmt\" (UniqueName: \"kubernetes.io/projected/30d6699e-f439-44ca-879b-4bb1267cd5d3-kube-api-access-vzjmt\") pod \"payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p\" (UID: \"30d6699e-f439-44ca-879b-4bb1267cd5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"
Oct 29 22:51:30.246668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.246643    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d6699e-f439-44ca-879b-4bb1267cd5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p\" (UID: \"30d6699e-f439-44ca-879b-4bb1267cd5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"
Oct 29 22:51:30.256839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.256742    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-3ee017-dockercfg-qpk8f\""
Oct 29 22:51:30.257186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.257152    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-3ee017\""
Oct 29 22:51:30.348453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.348414    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d6699e-f439-44ca-879b-4bb1267cd5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p\" (UID: \"30d6699e-f439-44ca-879b-4bb1267cd5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"
Oct 29 22:51:30.348917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.348861    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzjmt\" (UniqueName: \"kubernetes.io/projected/30d6699e-f439-44ca-879b-4bb1267cd5d3-kube-api-access-vzjmt\") pod \"payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p\" (UID: \"30d6699e-f439-44ca-879b-4bb1267cd5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"
Oct 29 22:51:30.350217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.350187    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d6699e-f439-44ca-879b-4bb1267cd5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p\" (UID: \"30d6699e-f439-44ca-879b-4bb1267cd5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"
Oct 29 22:51:30.371241 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.370914    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzjmt\" (UniqueName: \"kubernetes.io/projected/30d6699e-f439-44ca-879b-4bb1267cd5d3-kube-api-access-vzjmt\") pod \"payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p\" (UID: \"30d6699e-f439-44ca-879b-4bb1267cd5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"
Oct 29 22:51:30.470959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.470924    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"
Oct 29 22:51:30.779851 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:51:30.779814    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d6699e_f439_44ca_879b_4bb1267cd5d3.slice/crio-ea7d1d1342bec06a5fbb0fc2987c54e0ea1959d96edcc02fb597289e7624c48d WatchSource:0}: Error finding container ea7d1d1342bec06a5fbb0fc2987c54e0ea1959d96edcc02fb597289e7624c48d: Status 404 returned error can't find the container with id ea7d1d1342bec06a5fbb0fc2987c54e0ea1959d96edcc02fb597289e7624c48d
Oct 29 22:51:30.783466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.783442    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"]
Oct 29 22:51:30.836758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.836723    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:30.838069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:30.838041    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p" event={"ID":"30d6699e-f439-44ca-879b-4bb1267cd5d3","Type":"ContainerStarted","Data":"ea7d1d1342bec06a5fbb0fc2987c54e0ea1959d96edcc02fb597289e7624c48d"}
Oct 29 22:51:31.849351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:31.849277    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p" event={"ID":"30d6699e-f439-44ca-879b-4bb1267cd5d3","Type":"ContainerStarted","Data":"49d1b2f0c4025a1cb896a4f5301f5036ef83d21c571e8483d6b1932accf10c43"}
Oct 29 22:51:31.849732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:31.849360    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"
Oct 29 22:51:31.896421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:31.896376    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p" podStartSLOduration=1.896361802 podStartE2EDuration="1.896361802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:31.891483928 +0000 UTC m=+2233.916260059" watchObservedRunningTime="2026-10-29 22:51:31.896361802 +0000 UTC m=+2233.921137958"
Oct 29 22:51:32.481389 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:32.481354    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:34.355174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:34.355140    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:36.062514 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:36.062469    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:37.103407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.103377    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"]
Oct 29 22:51:37.108105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.108085    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.140524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.140476    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-06a95f-openshift-default-dockercfg-gb4cw\""
Oct 29 22:51:37.149268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.149237    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"]
Oct 29 22:51:37.252580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.252555    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-certs\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.252754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.252597    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-data\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.252754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.252723    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvqlg\" (UniqueName: \"kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-kube-api-access-fvqlg\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.252898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.252835    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-socket\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.252898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.252867    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-envoy\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.253000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.252980    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-token\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.253049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.253017    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-credential-socket\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.253108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.253049    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-podinfo\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.253173 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.253155    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istiod-ca-cert\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.354750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.354690    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvqlg\" (UniqueName: \"kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-kube-api-access-fvqlg\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.354880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.354828    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-socket\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.354880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.354846    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-envoy\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.354964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.354895    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-token\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.354964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.354933    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-credential-socket\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.355035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.354980    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-podinfo\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.355035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.355007    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istiod-ca-cert\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.355135 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.355118    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-certs\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.355192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.355168    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-data\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.355248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.355214    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-socket\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.355528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.355351    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-credential-socket\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.355640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.355583    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-certs\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.355708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.355659    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-data\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.356000 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.355968    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istiod-ca-cert\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.357954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.357932    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-envoy\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.358094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.358076    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-podinfo\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.375464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.375439    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-token\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.375672 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.375656    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvqlg\" (UniqueName: \"kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-kube-api-access-fvqlg\") pod \"e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") " pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.422800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.422779    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:37.738346 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:51:37.738304    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5143df2_3ef6_47a5_9c4c_87d0ac7a0346.slice/crio-8797dd4d727d60234b8f87acdf1ed795089256de4310bbb280b037557e3b3b21 WatchSource:0}: Error finding container 8797dd4d727d60234b8f87acdf1ed795089256de4310bbb280b037557e3b3b21: Status 404 returned error can't find the container with id 8797dd4d727d60234b8f87acdf1ed795089256de4310bbb280b037557e3b3b21
Oct 29 22:51:37.741168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.741129    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:51:37.741285 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.741203    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:51:37.741285 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.741233    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:51:37.741399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.741348    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"]
Oct 29 22:51:37.757488 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.757465    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:37.902802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.902762    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" event={"ID":"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346","Type":"ContainerStarted","Data":"1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc"}
Oct 29 22:51:37.903024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.902834    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" event={"ID":"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346","Type":"ContainerStarted","Data":"8797dd4d727d60234b8f87acdf1ed795089256de4310bbb280b037557e3b3b21"}
Oct 29 22:51:37.962530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:37.962481    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" podStartSLOduration=0.962466757 podStartE2EDuration="962.466757ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:37.959655739 +0000 UTC m=+2239.984431871" watchObservedRunningTime="2026-10-29 22:51:37.962466757 +0000 UTC m=+2239.987242888"
Oct 29 22:51:38.422972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:38.422934    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:38.424412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:38.424389    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:38.424548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:38.424437    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" podUID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 29 22:51:39.423881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:39.423847    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:39.424262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:39.423901    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" podUID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 29 22:51:39.577901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:39.577864    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:40.365269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.365238    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:40.439354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.438490    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:40.439354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.438552    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" podUID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 29 22:51:40.927744 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.927709    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"]
Oct 29 22:51:40.932643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.932614    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:40.944455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.944424    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee7a61c-5560-4470-8f87-04361f3c4804-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:40.944654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.944631    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:40.944772 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.944681    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:40.944772 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.944720    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:40.944936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.944920    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:40.944993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.944949    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:40.944993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.944981    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:40.945117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.945045    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:40.945117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.945097    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgsg\" (UniqueName: \"kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-kube-api-access-jvgsg\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:40.953588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.953564    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"]
Oct 29 22:51:40.959460 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:40.959440    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-d68c2c-openshift-default-dockercfg-28wt6\""
Oct 29 22:51:41.046336 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.046274    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee7a61c-5560-4470-8f87-04361f3c4804-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.046534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.046340    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.046534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.046374    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.046534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.046408    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.046710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.046548    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.046710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.046576    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.046710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.046610    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.046710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.046681    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.046929 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.046772    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgsg\" (UniqueName: \"kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-kube-api-access-jvgsg\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.046929 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.046789    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.047143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.047120    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.047403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.047378    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.047779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.047736    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.048053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.047767    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee7a61c-5560-4470-8f87-04361f3c4804-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.049364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.049341    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.049680 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.049658    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.076347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.067837    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgsg\" (UniqueName: \"kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-kube-api-access-jvgsg\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.076347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.075784    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.249171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.249139    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:41.260381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.260350    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:41.423940 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.423908    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:41.424084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.423970    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" podUID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 29 22:51:41.661809 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:51:41.661776    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee7a61c_5560_4470_8f87_04361f3c4804.slice/crio-7cbea702ff6bc3d0eb64180f15e3516d123a7c0cd25f37c1b8b84e4744e2a14b WatchSource:0}: Error finding container 7cbea702ff6bc3d0eb64180f15e3516d123a7c0cd25f37c1b8b84e4744e2a14b: Status 404 returned error can't find the container with id 7cbea702ff6bc3d0eb64180f15e3516d123a7c0cd25f37c1b8b84e4744e2a14b
Oct 29 22:51:41.665073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.665032    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:51:41.665174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.665131    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:51:41.665245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.665174    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:51:41.665364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.665344    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"]
Oct 29 22:51:41.940668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.940633    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7" event={"ID":"1ee7a61c-5560-4470-8f87-04361f3c4804","Type":"ContainerStarted","Data":"8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de"}
Oct 29 22:51:41.940860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.940694    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7" event={"ID":"1ee7a61c-5560-4470-8f87-04361f3c4804","Type":"ContainerStarted","Data":"7cbea702ff6bc3d0eb64180f15e3516d123a7c0cd25f37c1b8b84e4744e2a14b"}
Oct 29 22:51:41.999498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:41.999437    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7" podStartSLOduration=1.9994174550000001 podStartE2EDuration="1.999417455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:41.998715476 +0000 UTC m=+2244.023491608" watchObservedRunningTime="2026-10-29 22:51:41.999417455 +0000 UTC m=+2244.024193587"
Oct 29 22:51:42.250384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:42.250290    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:42.252270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:42.252247    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:42.252409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:42.252292    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7" podUID="1ee7a61c-5560-4470-8f87-04361f3c4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Oct 29 22:51:42.424452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:42.424419    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:42.424636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:42.424469    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" podUID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 29 22:51:42.860597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:42.860570    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"
Oct 29 22:51:43.025794 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:43.025666    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:43.249820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:43.249787    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:43.250043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:43.249832    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7" podUID="1ee7a61c-5560-4470-8f87-04361f3c4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Oct 29 22:51:43.423376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:43.423318    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:43.423535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:43.423407    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" podUID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 29 22:51:44.249699 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:44.249646    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:44.250078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:44.249727    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7" podUID="1ee7a61c-5560-4470-8f87-04361f3c4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Oct 29 22:51:44.490356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:44.489616    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:44.493359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:44.490941    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:44.497889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:44.497860    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:51:44.862648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:44.862615    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:45.249585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:45.249553    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:45.249741 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:45.249607    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7" podUID="1ee7a61c-5560-4470-8f87-04361f3c4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Oct 29 22:51:46.243570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:46.243538    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:46.249570 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:46.249542    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:46.249715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:46.249594    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7" podUID="1ee7a61c-5560-4470-8f87-04361f3c4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Oct 29 22:51:46.593271 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:46.592937    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:47.142616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:47.142575    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:47.250510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:47.250473    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Oct 29 22:51:47.250709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:47.250540    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7" podUID="1ee7a61c-5560-4470-8f87-04361f3c4804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Oct 29 22:51:47.599975 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:47.599945    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:47.743248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:47.743198    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:48.057762 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:51:48.057718    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:51:48.299643 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:48.299612    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:48.504792 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:48.504759    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:48.505530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:48.505498    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:48.506547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:48.506525    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 22:51:50.186603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:50.186572    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:51.641676 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:51.641636    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:51.772885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:51.772852    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:52.116596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:52.116566    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:52.240866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:52.240831    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:53.459593 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:53.459560    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:53.584585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:53.584552    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:54.271336 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:54.271291    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:56.094966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:56.094928    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:57.985415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:57.985379    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:59.778871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:51:59.778830    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:01.432645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:01.432584    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:03.273517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:03.273480    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:04.918742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:04.918711    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:06.754812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:06.754760    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:06.867679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:06.867645    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:06.981925 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:06.981890    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:08.568987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:08.568941    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:09.412392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.412354    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"]
Oct 29 22:52:09.418436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.418412    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.429371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.429345    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"]
Oct 29 22:52:09.433722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.433700    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.437874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.437848    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-f8ca96-openshift-default-dockercfg-ppssn\""
Oct 29 22:52:09.440681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.440658    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"]
Oct 29 22:52:09.459195 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.459168    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"]
Oct 29 22:52:09.551749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.551714    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.551943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.551760    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98fqc\" (UniqueName: \"kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-kube-api-access-98fqc\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.551943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.551830    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453867f4-909e-4d42-8ac8-6ab6da4f0626-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.551943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.551911    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.551943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.551930    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.552197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.551950    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.552197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.552095    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.552197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.552127    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffcbfcb-f8c9-441f-a456-b55829cad058-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.552379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.552213    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.552379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.552247    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.552379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.552269    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.552379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.552289    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.552379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.552353    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.552653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.552408    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.552653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.552456    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.552653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.552493    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwnp8\" (UniqueName: \"kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-kube-api-access-kwnp8\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.552653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.552560    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.552653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.552616    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.654310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.654278    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffcbfcb-f8c9-441f-a456-b55829cad058-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.654785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.654380    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.654785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.654416    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.654785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.654442    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.654785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.654474    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.654785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.654511    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.654785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.654542    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.654785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.654753    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.655180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.654874    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.655180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.654918    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.655180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655118    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.655180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655141    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffcbfcb-f8c9-441f-a456-b55829cad058-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.655433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655226    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwnp8\" (UniqueName: \"kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-kube-api-access-kwnp8\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.655433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655251    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.655433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655372    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.655433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655402    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.655661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655459    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.655661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655505    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98fqc\" (UniqueName: \"kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-kube-api-access-98fqc\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.655661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655542    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453867f4-909e-4d42-8ac8-6ab6da4f0626-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.655661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655573    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.655661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655597    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.655661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655623    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.655963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.655813    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.656397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.656373    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.656546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.656469    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.656735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.656715    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.656866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.656760    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453867f4-909e-4d42-8ac8-6ab6da4f0626-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.657156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.657134    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.657620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.657593    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.657750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.657728    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.657812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.657779    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.657892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.657875    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.676167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.676112    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwnp8\" (UniqueName: \"kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-kube-api-access-kwnp8\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.676281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.676195    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:09.676631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.676612    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.676856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.676837    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98fqc\" (UniqueName: \"kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-kube-api-access-98fqc\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.735050 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.735031    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:09.763013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:09.762992    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:10.058414 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:52:10.058366    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffcbfcb_f8c9_441f_a456_b55829cad058.slice/crio-25069c08207452a83899f6b4ac7a9529058e96d6ec4dc374d186be9fb0bc29e5 WatchSource:0}: Error finding container 25069c08207452a83899f6b4ac7a9529058e96d6ec4dc374d186be9fb0bc29e5: Status 404 returned error can't find the container with id 25069c08207452a83899f6b4ac7a9529058e96d6ec4dc374d186be9fb0bc29e5
Oct 29 22:52:10.062097 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.062052    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:52:10.062226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.062146    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:52:10.062226 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.062191    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:52:10.062388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.062193    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"]
Oct 29 22:52:10.102237 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.102209    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"]
Oct 29 22:52:10.105272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.105198    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:52:10.105428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.105373    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:52:10.105500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.105435    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:52:10.179034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.179005    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:10.299969 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.299931    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv" event={"ID":"bffcbfcb-f8c9-441f-a456-b55829cad058","Type":"ContainerStarted","Data":"5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58"}
Oct 29 22:52:10.300477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.300447    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv" event={"ID":"bffcbfcb-f8c9-441f-a456-b55829cad058","Type":"ContainerStarted","Data":"25069c08207452a83899f6b4ac7a9529058e96d6ec4dc374d186be9fb0bc29e5"}
Oct 29 22:52:10.303161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.303138    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm" event={"ID":"453867f4-909e-4d42-8ac8-6ab6da4f0626","Type":"ContainerStarted","Data":"ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b"}
Oct 29 22:52:10.303264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.303185    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm" event={"ID":"453867f4-909e-4d42-8ac8-6ab6da4f0626","Type":"ContainerStarted","Data":"23e2b0fbfcf7cfa517065bf86185c5eabfb48d94d5e557f2fe0e0330d27138e5"}
Oct 29 22:52:10.303734 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.303673    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:10.351348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.350755    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv" podStartSLOduration=1.350736318 podStartE2EDuration="1.350736318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:10.347752689 +0000 UTC m=+2272.372528822" watchObservedRunningTime="2026-10-29 22:52:10.350736318 +0000 UTC m=+2272.375512448"
Oct 29 22:52:10.735681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.735648    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:10.763937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:10.763907    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:11.737598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:11.737561    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 29 22:52:11.738047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:11.737622    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv" podUID="bffcbfcb-f8c9-441f-a456-b55829cad058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 29 22:52:11.764931 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:11.764900    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 29 22:52:11.765070 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:11.764946    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm" podUID="453867f4-909e-4d42-8ac8-6ab6da4f0626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 29 22:52:12.056082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:12.055957    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:12.735974 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:12.735939    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 29 22:52:12.736168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:12.735988    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv" podUID="bffcbfcb-f8c9-441f-a456-b55829cad058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 29 22:52:12.763854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:12.763826    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 29 22:52:12.764219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:12.763876    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm" podUID="453867f4-909e-4d42-8ac8-6ab6da4f0626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 29 22:52:13.733923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:13.733890    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:13.736071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:13.736043    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 29 22:52:13.736298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:13.736275    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv" podUID="bffcbfcb-f8c9-441f-a456-b55829cad058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 29 22:52:13.763733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:13.763693    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 29 22:52:13.763984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:13.763945    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm" podUID="453867f4-909e-4d42-8ac8-6ab6da4f0626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 29 22:52:13.859134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:13.859063    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:14.736043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:14.736011    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 29 22:52:14.736227 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:14.736056    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv" podUID="bffcbfcb-f8c9-441f-a456-b55829cad058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 29 22:52:14.764386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:14.764235    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 29 22:52:14.764386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:14.764275    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm" podUID="453867f4-909e-4d42-8ac8-6ab6da4f0626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 29 22:52:15.544065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:15.544028    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:15.736074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:15.736040    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 29 22:52:15.736254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:15.736091    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv" podUID="bffcbfcb-f8c9-441f-a456-b55829cad058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 29 22:52:15.763865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:15.763833    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 29 22:52:15.763977 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:15.763882    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm" podUID="453867f4-909e-4d42-8ac8-6ab6da4f0626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 29 22:52:16.735784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:16.735743    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body=
Oct 29 22:52:16.736183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:16.735820    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv" podUID="bffcbfcb-f8c9-441f-a456-b55829cad058" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused"
Oct 29 22:52:16.763520 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:16.763483    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 29 22:52:16.763693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:16.763548    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm" podUID="453867f4-909e-4d42-8ac8-6ab6da4f0626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 29 22:52:17.448708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:17.448665    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:17.740828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:17.740750    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:17.741282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:17.741234    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:17.741941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:17.741920    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 22:52:17.768153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:17.768123    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:17.768517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:17.768498    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:17.769229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:17.769214    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 22:52:17.789446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:17.789401    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm" podStartSLOduration=8.789388793 podStartE2EDuration="8.789388793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:10.426044418 +0000 UTC m=+2272.450820550" watchObservedRunningTime="2026-10-29 22:52:17.789388793 +0000 UTC m=+2279.814164924"
Oct 29 22:52:18.058620 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:52:18.058525    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:52:19.232398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:19.232367    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:20.950299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:20.950264    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:22.576776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:22.576734    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:23.316255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:23.316220    4860 scope.go:122] "RemoveContainer" containerID="a7bf51fab7b8cb122aaf4e9c5cb39f9db1091a73c240d60dc8cc9420ab06dcf1"
Oct 29 22:52:23.327857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:23.327835    4860 scope.go:122] "RemoveContainer" containerID="d2b8b743828eaa023976b1f29368b8179988b3c04d305a323e30770d60caa2af"
Oct 29 22:52:24.314986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:24.314944    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:26.124959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:26.124930    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:27.771426 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:27.771378    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:29.496080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:29.496050    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:31.288839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:31.288805    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:31.402689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:31.402656    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:31.523189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:31.523147    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:33.034511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:33.034476    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:34.673838 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:34.673808    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:36.638188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:36.638153    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:36.763528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:36.763489    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:38.029952 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:38.029919    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"]
Oct 29 22:52:38.043034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:38.043001    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df02d96-sx7nf"]
Oct 29 22:52:38.113618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:38.113589    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25bb3e8a-2427-41a7-bdbd-f1b922bd3734" path="/var/lib/kubelet/pods/25bb3e8a-2427-41a7-bdbd-f1b922bd3734/volumes"
Oct 29 22:52:38.369225 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:38.369140    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:38.588550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:38.588510    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:40.170619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:40.170582    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:41.822936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:41.822900    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:43.655752 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:43.655716    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:45.602505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:45.602470    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:46.615131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.615097    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"]
Oct 29 22:52:46.620006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.619984    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:46.637984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.637962    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"]
Oct 29 22:52:46.646166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.646137    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:52:46.646298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.646223    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:52:46.719134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.719099    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cltwt\" (UniqueName: \"kubernetes.io/projected/6e64b5e6-bae2-4768-bb6a-159914642d59-kube-api-access-cltwt\") pod \"maas-api-revoke-sub-4c89770e-sm5sl\" (UID: \"6e64b5e6-bae2-4768-bb6a-159914642d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:46.719314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.719232    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c89770e-sm5sl\" (UID: \"6e64b5e6-bae2-4768-bb6a-159914642d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:46.719410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.719375    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c89770e-sm5sl\" (UID: \"6e64b5e6-bae2-4768-bb6a-159914642d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:46.820905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.820874    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cltwt\" (UniqueName: \"kubernetes.io/projected/6e64b5e6-bae2-4768-bb6a-159914642d59-kube-api-access-cltwt\") pod \"maas-api-revoke-sub-4c89770e-sm5sl\" (UID: \"6e64b5e6-bae2-4768-bb6a-159914642d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:46.821077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.820983    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c89770e-sm5sl\" (UID: \"6e64b5e6-bae2-4768-bb6a-159914642d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:46.821077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.821037    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c89770e-sm5sl\" (UID: \"6e64b5e6-bae2-4768-bb6a-159914642d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:46.821832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.821808    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c89770e-sm5sl\" (UID: \"6e64b5e6-bae2-4768-bb6a-159914642d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:46.823544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.823523    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c89770e-sm5sl\" (UID: \"6e64b5e6-bae2-4768-bb6a-159914642d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:46.845075 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.845049    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cltwt\" (UniqueName: \"kubernetes.io/projected/6e64b5e6-bae2-4768-bb6a-159914642d59-kube-api-access-cltwt\") pod \"maas-api-revoke-sub-4c89770e-sm5sl\" (UID: \"6e64b5e6-bae2-4768-bb6a-159914642d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:46.934814 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:46.934791    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:47.239690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:47.239609    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:47.333395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:47.331838    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"]
Oct 29 22:52:47.334607 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:52:47.334579    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e64b5e6_bae2_4768_bb6a_159914642d59.slice/crio-bf73f5745f19f6183d24abbf4862666ea1f3fed28320a6160cab01f16a30418e WatchSource:0}: Error finding container bf73f5745f19f6183d24abbf4862666ea1f3fed28320a6160cab01f16a30418e: Status 404 returned error can't find the container with id bf73f5745f19f6183d24abbf4862666ea1f3fed28320a6160cab01f16a30418e
Oct 29 22:52:47.810470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:47.810434    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl" event={"ID":"6e64b5e6-bae2-4768-bb6a-159914642d59","Type":"ContainerStarted","Data":"bf73f5745f19f6183d24abbf4862666ea1f3fed28320a6160cab01f16a30418e"}
Oct 29 22:52:48.056885 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:52:48.056835    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:52:48.952058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:48.951165    4860 generic.go:356] "Generic (PLEG): container finished" podID="6e64b5e6-bae2-4768-bb6a-159914642d59" containerID="accdbfce51366d3fcbc23c355633d90970e8bf54d73ee1f8dbba5d6a3fe8af9c" exitCode=0
Oct 29 22:52:48.952058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:48.951242    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl" event={"ID":"6e64b5e6-bae2-4768-bb6a-159914642d59","Type":"ContainerDied","Data":"accdbfce51366d3fcbc23c355633d90970e8bf54d73ee1f8dbba5d6a3fe8af9c"}
Oct 29 22:52:49.101214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.101174    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:49.692027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.691987    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"]
Oct 29 22:52:49.707351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.704375    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:49.707351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.706222    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"]
Oct 29 22:52:49.798653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.798611    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48712fe8-jn9kx\" (UID: \"8a7a1462-6c14-4963-b9ee-246e986d876a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:49.798866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.798849    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnvm9\" (UniqueName: \"kubernetes.io/projected/8a7a1462-6c14-4963-b9ee-246e986d876a-kube-api-access-xnvm9\") pod \"maas-api-revoke-sub-48712fe8-jn9kx\" (UID: \"8a7a1462-6c14-4963-b9ee-246e986d876a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:49.798926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.798895    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48712fe8-jn9kx\" (UID: \"8a7a1462-6c14-4963-b9ee-246e986d876a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:49.900905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.900857    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnvm9\" (UniqueName: \"kubernetes.io/projected/8a7a1462-6c14-4963-b9ee-246e986d876a-kube-api-access-xnvm9\") pod \"maas-api-revoke-sub-48712fe8-jn9kx\" (UID: \"8a7a1462-6c14-4963-b9ee-246e986d876a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:49.900905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.900909    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48712fe8-jn9kx\" (UID: \"8a7a1462-6c14-4963-b9ee-246e986d876a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:49.901150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.901129    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48712fe8-jn9kx\" (UID: \"8a7a1462-6c14-4963-b9ee-246e986d876a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:49.901789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.901764    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48712fe8-jn9kx\" (UID: \"8a7a1462-6c14-4963-b9ee-246e986d876a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:49.903485 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.903465    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48712fe8-jn9kx\" (UID: \"8a7a1462-6c14-4963-b9ee-246e986d876a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:49.921912 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:49.921884    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnvm9\" (UniqueName: \"kubernetes.io/projected/8a7a1462-6c14-4963-b9ee-246e986d876a-kube-api-access-xnvm9\") pod \"maas-api-revoke-sub-48712fe8-jn9kx\" (UID: \"8a7a1462-6c14-4963-b9ee-246e986d876a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:50.025648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.025515    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:50.046973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.046943    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:50.235722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.235697    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:50.338421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.338386    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-openshift-ca\") pod \"6e64b5e6-bae2-4768-bb6a-159914642d59\" (UID: \"6e64b5e6-bae2-4768-bb6a-159914642d59\") "
Oct 29 22:52:50.338595 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.338438    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e64b5e6-bae2-4768-bb6a-159914642d59-kube-api-access-cltwt\" (UniqueName: \"kubernetes.io/projected/6e64b5e6-bae2-4768-bb6a-159914642d59-kube-api-access-cltwt\") pod \"6e64b5e6-bae2-4768-bb6a-159914642d59\" (UID: \"6e64b5e6-bae2-4768-bb6a-159914642d59\") "
Oct 29 22:52:50.338595 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.338562    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-xks-ca\") pod \"6e64b5e6-bae2-4768-bb6a-159914642d59\" (UID: \"6e64b5e6-bae2-4768-bb6a-159914642d59\") "
Oct 29 22:52:50.338825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.338796    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-openshift-ca" pod "6e64b5e6-bae2-4768-bb6a-159914642d59" (UID: "6e64b5e6-bae2-4768-bb6a-159914642d59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:50.339474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.339444    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:52:50.341148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.341116    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e64b5e6-bae2-4768-bb6a-159914642d59-kube-api-access-cltwt" pod "6e64b5e6-bae2-4768-bb6a-159914642d59" (UID: "6e64b5e6-bae2-4768-bb6a-159914642d59"). InnerVolumeSpecName "kube-api-access-cltwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:50.341302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.341247    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-xks-ca" pod "6e64b5e6-bae2-4768-bb6a-159914642d59" (UID: "6e64b5e6-bae2-4768-bb6a-159914642d59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:52:50.440788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.440758    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cltwt\" (UniqueName: \"kubernetes.io/projected/6e64b5e6-bae2-4768-bb6a-159914642d59-kube-api-access-cltwt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:52:50.440788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.440791    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e64b5e6-bae2-4768-bb6a-159914642d59-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:52:50.441996 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:52:50.441971    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7a1462_6c14_4963_b9ee_246e986d876a.slice/crio-c4f1c23685f3259586d02b15610e8ecb3f5257b202a702a092e280f16035adcb WatchSource:0}: Error finding container c4f1c23685f3259586d02b15610e8ecb3f5257b202a702a092e280f16035adcb: Status 404 returned error can't find the container with id c4f1c23685f3259586d02b15610e8ecb3f5257b202a702a092e280f16035adcb
Oct 29 22:52:50.444466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.444444    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"]
Oct 29 22:52:50.982125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.982090    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx" event={"ID":"8a7a1462-6c14-4963-b9ee-246e986d876a","Type":"ContainerStarted","Data":"86ed9538d7852b702bce3a419959f93ec4e0a03a67f6aefd3faf029d1304a327"}
Oct 29 22:52:50.982304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.982152    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx" event={"ID":"8a7a1462-6c14-4963-b9ee-246e986d876a","Type":"ContainerStarted","Data":"c4f1c23685f3259586d02b15610e8ecb3f5257b202a702a092e280f16035adcb"}
Oct 29 22:52:50.984189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.984152    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"
Oct 29 22:52:50.984189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.984169    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl" event={"ID":"6e64b5e6-bae2-4768-bb6a-159914642d59","Type":"ContainerDied","Data":"bf73f5745f19f6183d24abbf4862666ea1f3fed28320a6160cab01f16a30418e"}
Oct 29 22:52:50.984429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:50.984219    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf73f5745f19f6183d24abbf4862666ea1f3fed28320a6160cab01f16a30418e"
Oct 29 22:52:51.030234 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:51.030127    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx" podStartSLOduration=2.030112191 podStartE2EDuration="2.030112191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:51.025556268 +0000 UTC m=+2313.050332408" watchObservedRunningTime="2026-10-29 22:52:51.030112191 +0000 UTC m=+2313.054888322"
Oct 29 22:52:51.044442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:51.044415    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"]
Oct 29 22:52:51.048995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:51.048967    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1d3035-z7f7z"]
Oct 29 22:52:51.070891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:51.070851    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:51.996533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:51.996496    4860 generic.go:356] "Generic (PLEG): container finished" podID="8a7a1462-6c14-4963-b9ee-246e986d876a" containerID="86ed9538d7852b702bce3a419959f93ec4e0a03a67f6aefd3faf029d1304a327" exitCode=0
Oct 29 22:52:51.996733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:51.996589    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx" event={"ID":"8a7a1462-6c14-4963-b9ee-246e986d876a","Type":"ContainerDied","Data":"86ed9538d7852b702bce3a419959f93ec4e0a03a67f6aefd3faf029d1304a327"}
Oct 29 22:52:52.116001 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:52.115968    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a20b0cc5-7994-45bb-b2fe-f528be4ec7d6" path="/var/lib/kubelet/pods/a20b0cc5-7994-45bb-b2fe-f528be4ec7d6/volumes"
Oct 29 22:52:52.720557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:52.720515    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:53.210166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.210138    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:53.390917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.390823    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-openshift-ca\") pod \"8a7a1462-6c14-4963-b9ee-246e986d876a\" (UID: \"8a7a1462-6c14-4963-b9ee-246e986d876a\") "
Oct 29 22:52:53.390917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.390871    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-xks-ca\") pod \"8a7a1462-6c14-4963-b9ee-246e986d876a\" (UID: \"8a7a1462-6c14-4963-b9ee-246e986d876a\") "
Oct 29 22:52:53.391158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.391050    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7a1462-6c14-4963-b9ee-246e986d876a-kube-api-access-xnvm9\" (UniqueName: \"kubernetes.io/projected/8a7a1462-6c14-4963-b9ee-246e986d876a-kube-api-access-xnvm9\") pod \"8a7a1462-6c14-4963-b9ee-246e986d876a\" (UID: \"8a7a1462-6c14-4963-b9ee-246e986d876a\") "
Oct 29 22:52:53.391214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.391191    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-openshift-ca" pod "8a7a1462-6c14-4963-b9ee-246e986d876a" (UID: "8a7a1462-6c14-4963-b9ee-246e986d876a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:53.391677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.391653    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:52:53.393083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.393062    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-xks-ca" pod "8a7a1462-6c14-4963-b9ee-246e986d876a" (UID: "8a7a1462-6c14-4963-b9ee-246e986d876a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:52:53.393274 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.393255    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7a1462-6c14-4963-b9ee-246e986d876a-kube-api-access-xnvm9" pod "8a7a1462-6c14-4963-b9ee-246e986d876a" (UID: "8a7a1462-6c14-4963-b9ee-246e986d876a"). InnerVolumeSpecName "kube-api-access-xnvm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:53.493011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.492984    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7a1462-6c14-4963-b9ee-246e986d876a-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:52:53.493011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.493010    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnvm9\" (UniqueName: \"kubernetes.io/projected/8a7a1462-6c14-4963-b9ee-246e986d876a-kube-api-access-xnvm9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:52:53.699496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.699463    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"]
Oct 29 22:52:53.700665 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.700641    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8a7a1462-6c14-4963-b9ee-246e986d876a" containerName="revoke-keys"
Oct 29 22:52:53.700731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.700669    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7a1462-6c14-4963-b9ee-246e986d876a" containerName="revoke-keys"
Oct 29 22:52:53.700770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.700749    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6e64b5e6-bae2-4768-bb6a-159914642d59" containerName="revoke-keys"
Oct 29 22:52:53.700770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.700758    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e64b5e6-bae2-4768-bb6a-159914642d59" containerName="revoke-keys"
Oct 29 22:52:53.700986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.700970    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7a1462-6c14-4963-b9ee-246e986d876a" containerName="revoke-keys"
Oct 29 22:52:53.701023 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.701005    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e64b5e6-bae2-4768-bb6a-159914642d59" containerName="revoke-keys"
Oct 29 22:52:53.725762 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.725735    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"]
Oct 29 22:52:53.725916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.725842    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:53.898392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.898361    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmms\" (UniqueName: \"kubernetes.io/projected/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-kube-api-access-lxmms\") pod \"maas-api-revoke-sub-a41fb3d7-lt25l\" (UID: \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:53.898581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.898399    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a41fb3d7-lt25l\" (UID: \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:53.898581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:53.898422    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a41fb3d7-lt25l\" (UID: \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:54.000419 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.000313    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmms\" (UniqueName: \"kubernetes.io/projected/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-kube-api-access-lxmms\") pod \"maas-api-revoke-sub-a41fb3d7-lt25l\" (UID: \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:54.000419 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.000394    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a41fb3d7-lt25l\" (UID: \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:54.000618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.000419    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a41fb3d7-lt25l\" (UID: \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:54.001060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.001039    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a41fb3d7-lt25l\" (UID: \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:54.002828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.002804    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a41fb3d7-lt25l\" (UID: \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:54.015533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.015510    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"
Oct 29 22:52:54.015641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.015535    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx" event={"ID":"8a7a1462-6c14-4963-b9ee-246e986d876a","Type":"ContainerDied","Data":"c4f1c23685f3259586d02b15610e8ecb3f5257b202a702a092e280f16035adcb"}
Oct 29 22:52:54.015641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.015589    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4f1c23685f3259586d02b15610e8ecb3f5257b202a702a092e280f16035adcb"
Oct 29 22:52:54.021286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.021261    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmms\" (UniqueName: \"kubernetes.io/projected/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-kube-api-access-lxmms\") pod \"maas-api-revoke-sub-a41fb3d7-lt25l\" (UID: \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:54.058904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.058829    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:54.340703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.340676    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:54.365783 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.365755    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"]
Oct 29 22:52:54.546290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.546230    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"]
Oct 29 22:52:54.552141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.552122    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.612683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.612613    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"]
Oct 29 22:52:54.617127 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.617096    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.617251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.617139    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.617298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.617237    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.617394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.617371    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.617564 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.617545    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.617682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.617666    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.617729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.617703    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcr6s\" (UniqueName: \"kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-kube-api-access-gcr6s\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.617770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.617737    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.617835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.617819    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.718547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.718522    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.718686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.718587    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.718686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.718616    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcr6s\" (UniqueName: \"kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-kube-api-access-gcr6s\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.718686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.718637    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.718686 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.718675    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.718944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.718761    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.718944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.718783    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.718944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.718820    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.718944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.718848    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.719230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.719168    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-certs\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.719365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.719309    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-credential-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.719461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.719409    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-data\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.719691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.719666    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istiod-ca-cert\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.719800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.719713    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-socket\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.721150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.721131    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-envoy\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.721454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.721436    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-podinfo\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.739223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.739203    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcr6s\" (UniqueName: \"kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-kube-api-access-gcr6s\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.739223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.739214    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-token\") pod \"e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") " pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:54.893137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:54.893100    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:55.030278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.030240    4860 generic.go:356] "Generic (PLEG): container finished" podID="33379dd5-161b-4506-9860-228572c35c00" containerID="5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013" exitCode=0
Oct 29 22:52:55.030486 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.030304    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" event={"ID":"33379dd5-161b-4506-9860-228572c35c00","Type":"ContainerDied","Data":"5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013"}
Oct 29 22:52:55.030786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.030760    4860 scope.go:122] "RemoveContainer" containerID="5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013"
Oct 29 22:52:55.032828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.032806    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l" event={"ID":"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15","Type":"ContainerStarted","Data":"239e4c5afc9093163585e20440671a76e3ce6332178017beccaf65c7f8f26d09"}
Oct 29 22:52:55.032956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.032843    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l" event={"ID":"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15","Type":"ContainerStarted","Data":"22cc04d75398120b63448c02833e5d5f8629dc07a9cdcf7b033a5d8b3b4047ec"}
Oct 29 22:52:55.033187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.033147    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:52:55.033268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.033248    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:52:55.033386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.033306    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:52:55.133809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.133720    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l" podStartSLOduration=2.133706258 podStartE2EDuration="2.133706258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:55.120688709 +0000 UTC m=+2317.145464841" watchObservedRunningTime="2026-10-29 22:52:55.133706258 +0000 UTC m=+2317.158482388"
Oct 29 22:52:55.214942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.214904    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"]
Oct 29 22:52:55.217729 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:52:55.217703    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7b50c2_573d_41c0_b372_b51fd1e29ba6.slice/crio-c13c43be018afafae18507e4092cc6d1e8a815ce75f158e3e324be04b4d639a1 WatchSource:0}: Error finding container c13c43be018afafae18507e4092cc6d1e8a815ce75f158e3e324be04b4d639a1: Status 404 returned error can't find the container with id c13c43be018afafae18507e4092cc6d1e8a815ce75f158e3e324be04b4d639a1
Oct 29 22:52:55.222395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.221791    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:52:55.222395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.221886    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:52:55.222395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:55.221933    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:52:56.043013 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.042977    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b" event={"ID":"7c7b50c2-573d-41c0-b372-b51fd1e29ba6","Type":"ContainerStarted","Data":"20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878"}
Oct 29 22:52:56.043492 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.043034    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b" event={"ID":"7c7b50c2-573d-41c0-b372-b51fd1e29ba6","Type":"ContainerStarted","Data":"c13c43be018afafae18507e4092cc6d1e8a815ce75f158e3e324be04b4d639a1"}
Oct 29 22:52:56.045297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.045267    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" event={"ID":"33379dd5-161b-4506-9860-228572c35c00","Type":"ContainerStarted","Data":"1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301"}
Oct 29 22:52:56.046746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.046722    4860 generic.go:356] "Generic (PLEG): container finished" podID="fd1fb025-6a6e-46a3-98db-e36c4ed6ea15" containerID="239e4c5afc9093163585e20440671a76e3ce6332178017beccaf65c7f8f26d09" exitCode=0
Oct 29 22:52:56.046852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.046839    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l" event={"ID":"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15","Type":"ContainerDied","Data":"239e4c5afc9093163585e20440671a76e3ce6332178017beccaf65c7f8f26d09"}
Oct 29 22:52:56.109162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.109121    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b" podStartSLOduration=2.109107724 podStartE2EDuration="2.109107724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:56.103576934 +0000 UTC m=+2318.128353065" watchObservedRunningTime="2026-10-29 22:52:56.109107724 +0000 UTC m=+2318.133883855"
Oct 29 22:52:56.175167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.175122    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:56.816449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.814668    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:52:56.816449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.814801    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 29 22:52:56.816449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.814844    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 29 22:52:56.896129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.896092    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:52:56.896477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.896450    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 29 22:52:56.896685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.896659    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b" podUID="7c7b50c2-573d-41c0-b372-b51fd1e29ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 29 22:52:56.950345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.950280    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"]
Oct 29 22:52:56.955957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.955934    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:52:56.990352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:56.990072    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"]
Oct 29 22:52:57.064997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.064937    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00fa767c-tmmm9\" (UID: \"46558b20-00a1-4b08-a62f-889455b0e18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:52:57.065516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.065150    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00fa767c-tmmm9\" (UID: \"46558b20-00a1-4b08-a62f-889455b0e18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:52:57.065516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.065295    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpwz\" (UniqueName: \"kubernetes.io/projected/46558b20-00a1-4b08-a62f-889455b0e18f-kube-api-access-nxpwz\") pod \"maas-api-revoke-sub-00fa767c-tmmm9\" (UID: \"46558b20-00a1-4b08-a62f-889455b0e18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:52:57.166610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.166575    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00fa767c-tmmm9\" (UID: \"46558b20-00a1-4b08-a62f-889455b0e18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:52:57.166811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.166792    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00fa767c-tmmm9\" (UID: \"46558b20-00a1-4b08-a62f-889455b0e18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:52:57.167024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.167000    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpwz\" (UniqueName: \"kubernetes.io/projected/46558b20-00a1-4b08-a62f-889455b0e18f-kube-api-access-nxpwz\") pod \"maas-api-revoke-sub-00fa767c-tmmm9\" (UID: \"46558b20-00a1-4b08-a62f-889455b0e18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:52:57.167702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.167669    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00fa767c-tmmm9\" (UID: \"46558b20-00a1-4b08-a62f-889455b0e18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:52:57.232710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.229650    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpwz\" (UniqueName: \"kubernetes.io/projected/46558b20-00a1-4b08-a62f-889455b0e18f-kube-api-access-nxpwz\") pod \"maas-api-revoke-sub-00fa767c-tmmm9\" (UID: \"46558b20-00a1-4b08-a62f-889455b0e18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:52:57.232710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.232630    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00fa767c-tmmm9\" (UID: \"46558b20-00a1-4b08-a62f-889455b0e18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:52:57.321407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.313528    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:52:57.457556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.455717    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:57.630673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.630558    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-kube-api-access-lxmms\" (UniqueName: \"kubernetes.io/projected/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-kube-api-access-lxmms\") pod \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\" (UID: \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\") "
Oct 29 22:52:57.631162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.631111    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-xks-ca\") pod \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\" (UID: \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\") "
Oct 29 22:52:57.631479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.631434    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-openshift-ca\") pod \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\" (UID: \"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15\") "
Oct 29 22:52:57.632983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.632940    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-openshift-ca" pod "fd1fb025-6a6e-46a3-98db-e36c4ed6ea15" (UID: "fd1fb025-6a6e-46a3-98db-e36c4ed6ea15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:52:57.635513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.635491    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-xks-ca" pod "fd1fb025-6a6e-46a3-98db-e36c4ed6ea15" (UID: "fd1fb025-6a6e-46a3-98db-e36c4ed6ea15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:52:57.637864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.637836    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-kube-api-access-lxmms" pod "fd1fb025-6a6e-46a3-98db-e36c4ed6ea15" (UID: "fd1fb025-6a6e-46a3-98db-e36c4ed6ea15"). InnerVolumeSpecName "kube-api-access-lxmms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:52:57.737670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.737638    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:52:57.737852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.737836    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:52:57.737964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.737951    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxmms\" (UniqueName: \"kubernetes.io/projected/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15-kube-api-access-lxmms\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:52:57.738903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.738872    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"]
Oct 29 22:52:57.818346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.817281    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:52:57.818346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.817491    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 29 22:52:57.818346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.817541    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 29 22:52:57.895560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.895134    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 29 22:52:57.895560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.895192    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b" podUID="7c7b50c2-573d-41c0-b372-b51fd1e29ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 29 22:52:57.967881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:57.967843    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:58.154937 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:58.154909    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"
Oct 29 22:52:58.155414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:58.155392    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l" event={"ID":"fd1fb025-6a6e-46a3-98db-e36c4ed6ea15","Type":"ContainerDied","Data":"22cc04d75398120b63448c02833e5d5f8629dc07a9cdcf7b033a5d8b3b4047ec"}
Oct 29 22:52:58.155521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:58.155436    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22cc04d75398120b63448c02833e5d5f8629dc07a9cdcf7b033a5d8b3b4047ec"
Oct 29 22:52:58.157692 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:58.157669    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9" event={"ID":"46558b20-00a1-4b08-a62f-889455b0e18f","Type":"ContainerStarted","Data":"8f952cfcf9f246e8ad7bf04cbe7f92f1f378f7a84f2263f683c169fc22447096"}
Oct 29 22:52:58.818377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:58.816124    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 29 22:52:58.818377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:58.816184    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 29 22:52:58.897348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:58.895624    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 29 22:52:58.897348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:58.895678    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b" podUID="7c7b50c2-573d-41c0-b372-b51fd1e29ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 29 22:52:59.167401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:59.167272    4860 generic.go:356] "Generic (PLEG): container finished" podID="46558b20-00a1-4b08-a62f-889455b0e18f" containerID="429d69c2c24a09b65362123aa844aa4c70974243b5a728fc049acbde18b35fe7" exitCode=0
Oct 29 22:52:59.167865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:59.167445    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9" event={"ID":"46558b20-00a1-4b08-a62f-889455b0e18f","Type":"ContainerDied","Data":"429d69c2c24a09b65362123aa844aa4c70974243b5a728fc049acbde18b35fe7"}
Oct 29 22:52:59.764565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:59.764533    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:59.819906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:59.816683    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 29 22:52:59.819906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:59.816760    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 29 22:52:59.902359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:59.901714    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 29 22:52:59.902359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:52:59.901777    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b" podUID="7c7b50c2-573d-41c0-b372-b51fd1e29ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 29 22:53:00.025068 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.024980    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"]
Oct 29 22:53:00.025983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.025958    4860 cpu_manager.go:414] "containerMap: removing container" podUID="fd1fb025-6a6e-46a3-98db-e36c4ed6ea15" containerName="revoke-keys"
Oct 29 22:53:00.026192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.026175    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1fb025-6a6e-46a3-98db-e36c4ed6ea15" containerName="revoke-keys"
Oct 29 22:53:00.026568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.026552    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1fb025-6a6e-46a3-98db-e36c4ed6ea15" containerName="revoke-keys"
Oct 29 22:53:00.031648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.031629    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:00.033930 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.033521    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"]
Oct 29 22:53:00.131557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.131523    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e45f3e7-gw6p8\" (UID: \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:00.131781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.131587    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e45f3e7-gw6p8\" (UID: \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:00.131781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.131742    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf75t\" (UniqueName: \"kubernetes.io/projected/80ba7505-ead5-4b7e-b810-3d6f497b1e14-kube-api-access-qf75t\") pod \"maas-api-revoke-sub-2e45f3e7-gw6p8\" (UID: \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:00.233043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.233008    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf75t\" (UniqueName: \"kubernetes.io/projected/80ba7505-ead5-4b7e-b810-3d6f497b1e14-kube-api-access-qf75t\") pod \"maas-api-revoke-sub-2e45f3e7-gw6p8\" (UID: \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:00.233535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.233178    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e45f3e7-gw6p8\" (UID: \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:00.233535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.233233    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e45f3e7-gw6p8\" (UID: \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:00.234555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.234533    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e45f3e7-gw6p8\" (UID: \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:00.235809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.235790    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e45f3e7-gw6p8\" (UID: \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:00.255545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.255506    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf75t\" (UniqueName: \"kubernetes.io/projected/80ba7505-ead5-4b7e-b810-3d6f497b1e14-kube-api-access-qf75t\") pod \"maas-api-revoke-sub-2e45f3e7-gw6p8\" (UID: \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:00.351016 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.350981    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:00.526920 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.526877    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:53:00.540231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.539804    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46558b20-00a1-4b08-a62f-889455b0e18f-kube-api-access-nxpwz\" (UniqueName: \"kubernetes.io/projected/46558b20-00a1-4b08-a62f-889455b0e18f-kube-api-access-nxpwz\") pod \"46558b20-00a1-4b08-a62f-889455b0e18f\" (UID: \"46558b20-00a1-4b08-a62f-889455b0e18f\") "
Oct 29 22:53:00.540231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.540017    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-xks-ca\") pod \"46558b20-00a1-4b08-a62f-889455b0e18f\" (UID: \"46558b20-00a1-4b08-a62f-889455b0e18f\") "
Oct 29 22:53:00.540231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.540136    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-openshift-ca\") pod \"46558b20-00a1-4b08-a62f-889455b0e18f\" (UID: \"46558b20-00a1-4b08-a62f-889455b0e18f\") "
Oct 29 22:53:00.541593 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.541540    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-openshift-ca" pod "46558b20-00a1-4b08-a62f-889455b0e18f" (UID: "46558b20-00a1-4b08-a62f-889455b0e18f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:53:00.543072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.543027    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46558b20-00a1-4b08-a62f-889455b0e18f-kube-api-access-nxpwz" pod "46558b20-00a1-4b08-a62f-889455b0e18f" (UID: "46558b20-00a1-4b08-a62f-889455b0e18f"). InnerVolumeSpecName "kube-api-access-nxpwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:00.543978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.543954    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-xks-ca" pod "46558b20-00a1-4b08-a62f-889455b0e18f" (UID: "46558b20-00a1-4b08-a62f-889455b0e18f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:53:00.645157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.645071    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:00.645157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.645118    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46558b20-00a1-4b08-a62f-889455b0e18f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:00.645157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.645135    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxpwz\" (UniqueName: \"kubernetes.io/projected/46558b20-00a1-4b08-a62f-889455b0e18f-kube-api-access-nxpwz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:00.822839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.822736    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 29 22:53:00.822839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.822787    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 29 22:53:00.825134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.824500    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"]
Oct 29 22:53:00.899754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.899663    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 29 22:53:00.900024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:00.899983    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b" podUID="7c7b50c2-573d-41c0-b372-b51fd1e29ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 29 22:53:01.233508 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:01.233467    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8" event={"ID":"80ba7505-ead5-4b7e-b810-3d6f497b1e14","Type":"ContainerStarted","Data":"268c486d960d08862727b0b9e3cbd528cceebb5c569ac2150b03cfeba2f0e6b3"}
Oct 29 22:53:01.235070 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:01.235047    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"
Oct 29 22:53:01.235218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:01.235070    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9" event={"ID":"46558b20-00a1-4b08-a62f-889455b0e18f","Type":"ContainerDied","Data":"8f952cfcf9f246e8ad7bf04cbe7f92f1f378f7a84f2263f683c169fc22447096"}
Oct 29 22:53:01.235218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:01.235106    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f952cfcf9f246e8ad7bf04cbe7f92f1f378f7a84f2263f683c169fc22447096"
Oct 29 22:53:01.762854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:01.762805    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:01.814668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:01.813688    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body=
Oct 29 22:53:01.814668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:01.813746    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused"
Oct 29 22:53:01.901816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:01.901775    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body=
Oct 29 22:53:01.902022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:01.901831    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b" podUID="7c7b50c2-573d-41c0-b372-b51fd1e29ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused"
Oct 29 22:53:02.249207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:02.249168    4860 generic.go:356] "Generic (PLEG): container finished" podID="80ba7505-ead5-4b7e-b810-3d6f497b1e14" containerID="f2fbc458d58b3f95c3fea0c751965b391126db409598f2ea1d0b4d74df4de41f" exitCode=0
Oct 29 22:53:02.249750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:02.249254    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8" event={"ID":"80ba7505-ead5-4b7e-b810-3d6f497b1e14","Type":"ContainerDied","Data":"f2fbc458d58b3f95c3fea0c751965b391126db409598f2ea1d0b4d74df4de41f"}
Oct 29 22:53:02.816972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:02.816941    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:53:02.818108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:02.818086    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 22:53:02.897835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:02.897812    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:53:02.898106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:02.898086    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:53:02.898866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:02.898845    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 22:53:03.423331 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:03.423307    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:03.508086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:03.508051    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-xks-ca\") pod \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\" (UID: \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\") "
Oct 29 22:53:03.508423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:03.508118    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ba7505-ead5-4b7e-b810-3d6f497b1e14-kube-api-access-qf75t\" (UniqueName: \"kubernetes.io/projected/80ba7505-ead5-4b7e-b810-3d6f497b1e14-kube-api-access-qf75t\") pod \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\" (UID: \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\") "
Oct 29 22:53:03.508423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:03.508165    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-openshift-ca\") pod \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\" (UID: \"80ba7505-ead5-4b7e-b810-3d6f497b1e14\") "
Oct 29 22:53:03.509054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:03.509020    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-openshift-ca" pod "80ba7505-ead5-4b7e-b810-3d6f497b1e14" (UID: "80ba7505-ead5-4b7e-b810-3d6f497b1e14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:53:03.511266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:03.511236    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ba7505-ead5-4b7e-b810-3d6f497b1e14-kube-api-access-qf75t" pod "80ba7505-ead5-4b7e-b810-3d6f497b1e14" (UID: "80ba7505-ead5-4b7e-b810-3d6f497b1e14"). InnerVolumeSpecName "kube-api-access-qf75t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:03.514473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:03.514444    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-xks-ca" pod "80ba7505-ead5-4b7e-b810-3d6f497b1e14" (UID: "80ba7505-ead5-4b7e-b810-3d6f497b1e14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:53:03.549361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:03.549294    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:03.616622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:03.612551    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:03.616622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:03.612590    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf75t\" (UniqueName: \"kubernetes.io/projected/80ba7505-ead5-4b7e-b810-3d6f497b1e14-kube-api-access-qf75t\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:03.616622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:03.612607    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80ba7505-ead5-4b7e-b810-3d6f497b1e14-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:04.008266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.008227    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"]
Oct 29 22:53:04.009391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.009365    4860 cpu_manager.go:414] "containerMap: removing container" podUID="46558b20-00a1-4b08-a62f-889455b0e18f" containerName="revoke-keys"
Oct 29 22:53:04.009391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.009389    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46558b20-00a1-4b08-a62f-889455b0e18f" containerName="revoke-keys"
Oct 29 22:53:04.009597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.009427    4860 cpu_manager.go:414] "containerMap: removing container" podUID="80ba7505-ead5-4b7e-b810-3d6f497b1e14" containerName="revoke-keys"
Oct 29 22:53:04.009597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.009436    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ba7505-ead5-4b7e-b810-3d6f497b1e14" containerName="revoke-keys"
Oct 29 22:53:04.009717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.009671    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="46558b20-00a1-4b08-a62f-889455b0e18f" containerName="revoke-keys"
Oct 29 22:53:04.009717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.009694    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ba7505-ead5-4b7e-b810-3d6f497b1e14" containerName="revoke-keys"
Oct 29 22:53:04.014498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.014478    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:04.022265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.022237    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"]
Oct 29 22:53:04.119954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.119918    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e65e61-q2ccn\" (UID: \"d2083e44-42fb-4da8-b978-ee3a76aea574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:04.120149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.119963    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e65e61-q2ccn\" (UID: \"d2083e44-42fb-4da8-b978-ee3a76aea574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:04.120253 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.120230    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fztjg\" (UniqueName: \"kubernetes.io/projected/d2083e44-42fb-4da8-b978-ee3a76aea574-kube-api-access-fztjg\") pod \"maas-api-revoke-sub-48e65e61-q2ccn\" (UID: \"d2083e44-42fb-4da8-b978-ee3a76aea574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:04.221835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.221801    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztjg\" (UniqueName: \"kubernetes.io/projected/d2083e44-42fb-4da8-b978-ee3a76aea574-kube-api-access-fztjg\") pod \"maas-api-revoke-sub-48e65e61-q2ccn\" (UID: \"d2083e44-42fb-4da8-b978-ee3a76aea574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:04.222004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.221918    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e65e61-q2ccn\" (UID: \"d2083e44-42fb-4da8-b978-ee3a76aea574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:04.222004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.221945    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e65e61-q2ccn\" (UID: \"d2083e44-42fb-4da8-b978-ee3a76aea574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:04.222660 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.222632    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e65e61-q2ccn\" (UID: \"d2083e44-42fb-4da8-b978-ee3a76aea574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:04.224247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.224227    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e65e61-q2ccn\" (UID: \"d2083e44-42fb-4da8-b978-ee3a76aea574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:04.242159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.242138    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztjg\" (UniqueName: \"kubernetes.io/projected/d2083e44-42fb-4da8-b978-ee3a76aea574-kube-api-access-fztjg\") pod \"maas-api-revoke-sub-48e65e61-q2ccn\" (UID: \"d2083e44-42fb-4da8-b978-ee3a76aea574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:04.267380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.267303    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"
Oct 29 22:53:04.267380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.267364    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8" event={"ID":"80ba7505-ead5-4b7e-b810-3d6f497b1e14","Type":"ContainerDied","Data":"268c486d960d08862727b0b9e3cbd528cceebb5c569ac2150b03cfeba2f0e6b3"}
Oct 29 22:53:04.267500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.267410    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="268c486d960d08862727b0b9e3cbd528cceebb5c569ac2150b03cfeba2f0e6b3"
Oct 29 22:53:04.335468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.335448    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:04.634380 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:53:04.634351    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2083e44_42fb_4da8_b978_ee3a76aea574.slice/crio-32001ebdc2f2a834b32159e955fa4ba47e4cf31c2feb94cfc70cf75dff6bc593 WatchSource:0}: Error finding container 32001ebdc2f2a834b32159e955fa4ba47e4cf31c2feb94cfc70cf75dff6bc593: Status 404 returned error can't find the container with id 32001ebdc2f2a834b32159e955fa4ba47e4cf31c2feb94cfc70cf75dff6bc593
Oct 29 22:53:04.639380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:04.639350    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"]
Oct 29 22:53:05.277782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:05.277745    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn" event={"ID":"d2083e44-42fb-4da8-b978-ee3a76aea574","Type":"ContainerStarted","Data":"616fab32461c552f2c8b97d578bc0994b7b05c70226699949aa9f49214872ddb"}
Oct 29 22:53:05.277978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:05.277810    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn" event={"ID":"d2083e44-42fb-4da8-b978-ee3a76aea574","Type":"ContainerStarted","Data":"32001ebdc2f2a834b32159e955fa4ba47e4cf31c2feb94cfc70cf75dff6bc593"}
Oct 29 22:53:05.278817 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:05.278791    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:05.333997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:05.333588    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn" podStartSLOduration=2.333566682 podStartE2EDuration="2.333566682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:05.327502414 +0000 UTC m=+2327.352278549" watchObservedRunningTime="2026-10-29 22:53:05.333566682 +0000 UTC m=+2327.358342807"
Oct 29 22:53:06.287902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:06.287869    4860 generic.go:356] "Generic (PLEG): container finished" podID="d2083e44-42fb-4da8-b978-ee3a76aea574" containerID="616fab32461c552f2c8b97d578bc0994b7b05c70226699949aa9f49214872ddb" exitCode=0
Oct 29 22:53:06.288294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:06.287954    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn" event={"ID":"d2083e44-42fb-4da8-b978-ee3a76aea574","Type":"ContainerDied","Data":"616fab32461c552f2c8b97d578bc0994b7b05c70226699949aa9f49214872ddb"}
Oct 29 22:53:07.129594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:07.129562    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:07.478316 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:07.478288    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:07.582515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:07.582482    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2083e44-42fb-4da8-b978-ee3a76aea574-kube-api-access-fztjg\" (UniqueName: \"kubernetes.io/projected/d2083e44-42fb-4da8-b978-ee3a76aea574-kube-api-access-fztjg\") pod \"d2083e44-42fb-4da8-b978-ee3a76aea574\" (UID: \"d2083e44-42fb-4da8-b978-ee3a76aea574\") "
Oct 29 22:53:07.582691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:07.582575    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-xks-ca\") pod \"d2083e44-42fb-4da8-b978-ee3a76aea574\" (UID: \"d2083e44-42fb-4da8-b978-ee3a76aea574\") "
Oct 29 22:53:07.582756 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:07.582715    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-openshift-ca\") pod \"d2083e44-42fb-4da8-b978-ee3a76aea574\" (UID: \"d2083e44-42fb-4da8-b978-ee3a76aea574\") "
Oct 29 22:53:07.583172 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:07.583140    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-openshift-ca" pod "d2083e44-42fb-4da8-b978-ee3a76aea574" (UID: "d2083e44-42fb-4da8-b978-ee3a76aea574"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:53:07.584754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:07.584724    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-xks-ca" pod "d2083e44-42fb-4da8-b978-ee3a76aea574" (UID: "d2083e44-42fb-4da8-b978-ee3a76aea574"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:53:07.584882 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:07.584865    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2083e44-42fb-4da8-b978-ee3a76aea574-kube-api-access-fztjg" pod "d2083e44-42fb-4da8-b978-ee3a76aea574" (UID: "d2083e44-42fb-4da8-b978-ee3a76aea574"). InnerVolumeSpecName "kube-api-access-fztjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:07.684122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:07.684081    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:07.684122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:07.684122    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fztjg\" (UniqueName: \"kubernetes.io/projected/d2083e44-42fb-4da8-b978-ee3a76aea574-kube-api-access-fztjg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:07.684273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:07.684132    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2083e44-42fb-4da8-b978-ee3a76aea574-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:08.317374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:08.317336    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn" event={"ID":"d2083e44-42fb-4da8-b978-ee3a76aea574","Type":"ContainerDied","Data":"32001ebdc2f2a834b32159e955fa4ba47e4cf31c2feb94cfc70cf75dff6bc593"}
Oct 29 22:53:08.317374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:08.317364    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"
Oct 29 22:53:08.317585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:08.317396    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32001ebdc2f2a834b32159e955fa4ba47e4cf31c2feb94cfc70cf75dff6bc593"
Oct 29 22:53:08.848310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:08.848282    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:09.031396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:09.031361    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"]
Oct 29 22:53:09.044614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:09.044589    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b819be94-485nj"]
Oct 29 22:53:10.032841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:10.032809    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"]
Oct 29 22:53:10.046681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:10.046652    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406a9910-blsvx"]
Oct 29 22:53:10.114224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:10.114185    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0551700-3fdf-4bee-b534-0bd0a66278dc" path="/var/lib/kubelet/pods/c0551700-3fdf-4bee-b534-0bd0a66278dc/volumes"
Oct 29 22:53:10.114759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:10.114736    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8dfa7b8-e581-43df-9851-0e57408b78c1" path="/var/lib/kubelet/pods/d8dfa7b8-e581-43df-9851-0e57408b78c1/volumes"
Oct 29 22:53:10.502490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:10.502430    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:11.031783 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:11.031748    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"]
Oct 29 22:53:11.045368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:11.045306    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-309f2427-kzvk8"]
Oct 29 22:53:12.113949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:12.113915    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ae10b2-3f73-4b33-a431-069ef827f0f8" path="/var/lib/kubelet/pods/33ae10b2-3f73-4b33-a431-069ef827f0f8/volumes"
Oct 29 22:53:12.312087 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:12.312052    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:13.991842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:13.991804    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:14.159152 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:14.159090    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:14.298798 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:14.298720    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:15.141177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.141146    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"]
Oct 29 22:53:15.141897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.141880    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d2083e44-42fb-4da8-b978-ee3a76aea574" containerName="revoke-keys"
Oct 29 22:53:15.141957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.141897    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2083e44-42fb-4da8-b978-ee3a76aea574" containerName="revoke-keys"
Oct 29 22:53:15.142084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.142072    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2083e44-42fb-4da8-b978-ee3a76aea574" containerName="revoke-keys"
Oct 29 22:53:15.145803 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.145788    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:15.178342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.175055    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:53:15.178709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.178680    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:53:15.184796 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.184768    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"]
Oct 29 22:53:15.276666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.276636    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74mkq\" (UniqueName: \"kubernetes.io/projected/8b1bbd5d-f28c-4f80-ace9-52820f32f954-kube-api-access-74mkq\") pod \"maas-api-revoke-sub-d0245f65-sgj2w\" (UID: \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:15.276823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.276693    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0245f65-sgj2w\" (UID: \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:15.276881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.276865    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0245f65-sgj2w\" (UID: \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:15.378564 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.378530    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0245f65-sgj2w\" (UID: \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:15.378718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.378686    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mkq\" (UniqueName: \"kubernetes.io/projected/8b1bbd5d-f28c-4f80-ace9-52820f32f954-kube-api-access-74mkq\") pod \"maas-api-revoke-sub-d0245f65-sgj2w\" (UID: \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:15.378764 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.378734    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0245f65-sgj2w\" (UID: \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:15.379403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.379380    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0245f65-sgj2w\" (UID: \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:15.381089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.381065    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0245f65-sgj2w\" (UID: \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:15.399337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.399277    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74mkq\" (UniqueName: \"kubernetes.io/projected/8b1bbd5d-f28c-4f80-ace9-52820f32f954-kube-api-access-74mkq\") pod \"maas-api-revoke-sub-d0245f65-sgj2w\" (UID: \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:15.462789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.462767    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:15.765596 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:53:15.765564    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1bbd5d_f28c_4f80_ace9_52820f32f954.slice/crio-c46fa9dd03702ae818d78b99fcbb3a7521a50378c8fe61dbb0f7771aa4c0c05a WatchSource:0}: Error finding container c46fa9dd03702ae818d78b99fcbb3a7521a50378c8fe61dbb0f7771aa4c0c05a: Status 404 returned error can't find the container with id c46fa9dd03702ae818d78b99fcbb3a7521a50378c8fe61dbb0f7771aa4c0c05a
Oct 29 22:53:15.770927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:15.770899    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"]
Oct 29 22:53:16.266416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:16.266386    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:16.466697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:16.466661    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w" event={"ID":"8b1bbd5d-f28c-4f80-ace9-52820f32f954","Type":"ContainerStarted","Data":"a4e44ade94e3efc65dd161c7790e1364afe25fd4d99599f77a3b7c856bf1eda6"}
Oct 29 22:53:16.466873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:16.466723    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w" event={"ID":"8b1bbd5d-f28c-4f80-ace9-52820f32f954","Type":"ContainerStarted","Data":"c46fa9dd03702ae818d78b99fcbb3a7521a50378c8fe61dbb0f7771aa4c0c05a"}
Oct 29 22:53:16.508943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:16.508894    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w" podStartSLOduration=1.5088817190000001 podStartE2EDuration="1.508881719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:16.507622518 +0000 UTC m=+2338.532398669" watchObservedRunningTime="2026-10-29 22:53:16.508881719 +0000 UTC m=+2338.533657849"
Oct 29 22:53:17.097183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:17.097114    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:17.217163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:17.217123    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:17.476954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:17.476922    4860 generic.go:356] "Generic (PLEG): container finished" podID="8b1bbd5d-f28c-4f80-ace9-52820f32f954" containerID="a4e44ade94e3efc65dd161c7790e1364afe25fd4d99599f77a3b7c856bf1eda6" exitCode=0
Oct 29 22:53:17.477410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:17.477004    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w" event={"ID":"8b1bbd5d-f28c-4f80-ace9-52820f32f954","Type":"ContainerDied","Data":"a4e44ade94e3efc65dd161c7790e1364afe25fd4d99599f77a3b7c856bf1eda6"}
Oct 29 22:53:17.989564 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:17.989534    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:18.077436 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:53:18.077394    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:53:18.688037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.687982    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:18.694088 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.694065    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:18.817677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.817571    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:18.843831 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.843805    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-openshift-ca\") pod \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\" (UID: \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\") "
Oct 29 22:53:18.843947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.843890    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-xks-ca\") pod \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\" (UID: \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\") "
Oct 29 22:53:18.844168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.844148    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1bbd5d-f28c-4f80-ace9-52820f32f954-kube-api-access-74mkq\" (UniqueName: \"kubernetes.io/projected/8b1bbd5d-f28c-4f80-ace9-52820f32f954-kube-api-access-74mkq\") pod \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\" (UID: \"8b1bbd5d-f28c-4f80-ace9-52820f32f954\") "
Oct 29 22:53:18.844221 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.844191    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-openshift-ca" pod "8b1bbd5d-f28c-4f80-ace9-52820f32f954" (UID: "8b1bbd5d-f28c-4f80-ace9-52820f32f954"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:53:18.845366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.844738    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:18.846281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.846254    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-xks-ca" pod "8b1bbd5d-f28c-4f80-ace9-52820f32f954" (UID: "8b1bbd5d-f28c-4f80-ace9-52820f32f954"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:53:18.846890 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.846852    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1bbd5d-f28c-4f80-ace9-52820f32f954-kube-api-access-74mkq" pod "8b1bbd5d-f28c-4f80-ace9-52820f32f954" (UID: "8b1bbd5d-f28c-4f80-ace9-52820f32f954"). InnerVolumeSpecName "kube-api-access-74mkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:18.944701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.944660    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:18.946249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.946222    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74mkq\" (UniqueName: \"kubernetes.io/projected/8b1bbd5d-f28c-4f80-ace9-52820f32f954-kube-api-access-74mkq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:18.946369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:18.946252    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1bbd5d-f28c-4f80-ace9-52820f32f954-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:19.496138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:19.496110    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"
Oct 29 22:53:19.496345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:19.496152    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w" event={"ID":"8b1bbd5d-f28c-4f80-ace9-52820f32f954","Type":"ContainerDied","Data":"c46fa9dd03702ae818d78b99fcbb3a7521a50378c8fe61dbb0f7771aa4c0c05a"}
Oct 29 22:53:19.496345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:19.496236    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c46fa9dd03702ae818d78b99fcbb3a7521a50378c8fe61dbb0f7771aa4c0c05a"
Oct 29 22:53:19.963875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:19.963844    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:21.759875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:21.759841    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:23.366515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:23.366485    4860 scope.go:122] "RemoveContainer" containerID="cc3fb3c2570b4a72187439268d5b40ee34ee2434ebd6d142646bb94f8e8b4642"
Oct 29 22:53:23.378107 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:23.378079    4860 scope.go:122] "RemoveContainer" containerID="119e83b5640517bf5773622836980e0aa4a7321f77eb046adaf0715679aa4b26"
Oct 29 22:53:23.388006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:23.387985    4860 scope.go:122] "RemoveContainer" containerID="3cab11e6a7627b32ef3e8c9cd4143658c3a9500b9c0a29eb592b2eeae23beb7d"
Oct 29 22:53:23.423888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:23.423865    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:24.030786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:24.030754    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"]
Oct 29 22:53:24.043725 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:24.043698    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c716360b-x26bw"]
Oct 29 22:53:24.114267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:24.114230    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87d833d0-c4c3-4e18-809e-4ec1a1965117" path="/var/lib/kubelet/pods/87d833d0-c4c3-4e18-809e-4ec1a1965117/volumes"
Oct 29 22:53:24.691406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:24.691371    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:25.204541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.204508    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:25.613199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.613129    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"]
Oct 29 22:53:25.613861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.613845    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8b1bbd5d-f28c-4f80-ace9-52820f32f954" containerName="revoke-keys"
Oct 29 22:53:25.613915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.613863    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1bbd5d-f28c-4f80-ace9-52820f32f954" containerName="revoke-keys"
Oct 29 22:53:25.614021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.614012    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1bbd5d-f28c-4f80-ace9-52820f32f954" containerName="revoke-keys"
Oct 29 22:53:25.619392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.619374    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:25.637493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.637468    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"]
Oct 29 22:53:25.645091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.645061    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:53:25.645220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.645128    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:53:25.789740 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.789703    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bdc36f4-w5gsd\" (UID: \"289a6c0a-140b-471a-a549-cf3f2d07dabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:25.790240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.789781    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bdc36f4-w5gsd\" (UID: \"289a6c0a-140b-471a-a549-cf3f2d07dabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:25.790240 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.789955    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7pzv\" (UniqueName: \"kubernetes.io/projected/289a6c0a-140b-471a-a549-cf3f2d07dabc-kube-api-access-g7pzv\") pod \"maas-api-revoke-sub-8bdc36f4-w5gsd\" (UID: \"289a6c0a-140b-471a-a549-cf3f2d07dabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:25.856651 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.856621    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:25.898397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.891800    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bdc36f4-w5gsd\" (UID: \"289a6c0a-140b-471a-a549-cf3f2d07dabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:25.898397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.891898    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bdc36f4-w5gsd\" (UID: \"289a6c0a-140b-471a-a549-cf3f2d07dabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:25.898397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.892650    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7pzv\" (UniqueName: \"kubernetes.io/projected/289a6c0a-140b-471a-a549-cf3f2d07dabc-kube-api-access-g7pzv\") pod \"maas-api-revoke-sub-8bdc36f4-w5gsd\" (UID: \"289a6c0a-140b-471a-a549-cf3f2d07dabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:25.898397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.894016    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bdc36f4-w5gsd\" (UID: \"289a6c0a-140b-471a-a549-cf3f2d07dabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:25.950401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.936877    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bdc36f4-w5gsd\" (UID: \"289a6c0a-140b-471a-a549-cf3f2d07dabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:25.972459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:25.965093    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7pzv\" (UniqueName: \"kubernetes.io/projected/289a6c0a-140b-471a-a549-cf3f2d07dabc-kube-api-access-g7pzv\") pod \"maas-api-revoke-sub-8bdc36f4-w5gsd\" (UID: \"289a6c0a-140b-471a-a549-cf3f2d07dabc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:26.235880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:26.235854    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:26.543302 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:53:26.543274    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289a6c0a_140b_471a_a549_cf3f2d07dabc.slice/crio-57b7885c431554c720745976fb3d466a9f90491bee88b100ede3db86d91b1ff2 WatchSource:0}: Error finding container 57b7885c431554c720745976fb3d466a9f90491bee88b100ede3db86d91b1ff2: Status 404 returned error can't find the container with id 57b7885c431554c720745976fb3d466a9f90491bee88b100ede3db86d91b1ff2
Oct 29 22:53:26.547104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:26.547081    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"]
Oct 29 22:53:26.559170 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:26.559145    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd" event={"ID":"289a6c0a-140b-471a-a549-cf3f2d07dabc","Type":"ContainerStarted","Data":"57b7885c431554c720745976fb3d466a9f90491bee88b100ede3db86d91b1ff2"}
Oct 29 22:53:27.034535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:27.034505    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:27.576043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:27.575986    4860 generic.go:356] "Generic (PLEG): container finished" podID="289a6c0a-140b-471a-a549-cf3f2d07dabc" containerID="4aeb738f503c7209ad2c06d76a63597ecc5b2423508e2dc44a5e414e5c5bde66" exitCode=0
Oct 29 22:53:27.576517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:27.576490    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd" event={"ID":"289a6c0a-140b-471a-a549-cf3f2d07dabc","Type":"ContainerDied","Data":"4aeb738f503c7209ad2c06d76a63597ecc5b2423508e2dc44a5e414e5c5bde66"}
Oct 29 22:53:28.660396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:28.660369    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:28.795185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:28.795161    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:28.842113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:28.842051    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-xks-ca\") pod \"289a6c0a-140b-471a-a549-cf3f2d07dabc\" (UID: \"289a6c0a-140b-471a-a549-cf3f2d07dabc\") "
Oct 29 22:53:28.842265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:28.842245    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289a6c0a-140b-471a-a549-cf3f2d07dabc-kube-api-access-g7pzv\" (UniqueName: \"kubernetes.io/projected/289a6c0a-140b-471a-a549-cf3f2d07dabc-kube-api-access-g7pzv\") pod \"289a6c0a-140b-471a-a549-cf3f2d07dabc\" (UID: \"289a6c0a-140b-471a-a549-cf3f2d07dabc\") "
Oct 29 22:53:28.842353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:28.842295    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-openshift-ca\") pod \"289a6c0a-140b-471a-a549-cf3f2d07dabc\" (UID: \"289a6c0a-140b-471a-a549-cf3f2d07dabc\") "
Oct 29 22:53:28.842744 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:28.842711    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-openshift-ca" pod "289a6c0a-140b-471a-a549-cf3f2d07dabc" (UID: "289a6c0a-140b-471a-a549-cf3f2d07dabc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:53:28.842984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:28.842946    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:28.844455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:28.844428    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-xks-ca" pod "289a6c0a-140b-471a-a549-cf3f2d07dabc" (UID: "289a6c0a-140b-471a-a549-cf3f2d07dabc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:53:28.844603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:28.844579    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289a6c0a-140b-471a-a549-cf3f2d07dabc-kube-api-access-g7pzv" pod "289a6c0a-140b-471a-a549-cf3f2d07dabc" (UID: "289a6c0a-140b-471a-a549-cf3f2d07dabc"). InnerVolumeSpecName "kube-api-access-g7pzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:53:28.944129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:28.944098    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289a6c0a-140b-471a-a549-cf3f2d07dabc-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:28.944129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:28.944127    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7pzv\" (UniqueName: \"kubernetes.io/projected/289a6c0a-140b-471a-a549-cf3f2d07dabc-kube-api-access-g7pzv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:53:29.596395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:29.596365    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"
Oct 29 22:53:29.596395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:29.596376    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd" event={"ID":"289a6c0a-140b-471a-a549-cf3f2d07dabc","Type":"ContainerDied","Data":"57b7885c431554c720745976fb3d466a9f90491bee88b100ede3db86d91b1ff2"}
Oct 29 22:53:29.596678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:29.596421    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57b7885c431554c720745976fb3d466a9f90491bee88b100ede3db86d91b1ff2"
Oct 29 22:53:29.692477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:29.692440    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:30.474974 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:30.474941    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:32.236589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:32.236556    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:33.966381 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:33.966350    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:34.056211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:34.056176    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"]
Oct 29 22:53:34.073469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:34.073420    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"]
Oct 29 22:53:34.079165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:34.079128    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a01097d0-66jxx"]
Oct 29 22:53:34.087237 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:34.087205    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5f8d01-kkgnp"]
Oct 29 22:53:34.117461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:34.117427    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d4495a-f7e6-49ea-9a23-904a01a926e6" path="/var/lib/kubelet/pods/a0d4495a-f7e6-49ea-9a23-904a01a926e6/volumes"
Oct 29 22:53:34.118241 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:34.118220    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10" path="/var/lib/kubelet/pods/c8c6d23b-e3ce-4bf3-91e2-ecd2fd291a10/volumes"
Oct 29 22:53:34.133688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:34.133659    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:35.834459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:35.834426    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:37.524203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:37.524167    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:38.027463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:38.027428    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"]
Oct 29 22:53:38.040846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:38.040811    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa82ad1b-dmkx6"]
Oct 29 22:53:38.113728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:38.113699    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a25f99-c761-4c76-87c6-c8ace5698b17" path="/var/lib/kubelet/pods/c0a25f99-c761-4c76-87c6-c8ace5698b17/volumes"
Oct 29 22:53:38.840504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:38.840466    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:39.262314 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:39.262271    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:39.466604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:39.466572    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:39.923350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:39.919494    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"]
Oct 29 22:53:39.930360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:39.929581    4860 cpu_manager.go:414] "containerMap: removing container" podUID="289a6c0a-140b-471a-a549-cf3f2d07dabc" containerName="revoke-keys"
Oct 29 22:53:39.930360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:39.929613    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289a6c0a-140b-471a-a549-cf3f2d07dabc" containerName="revoke-keys"
Oct 29 22:53:39.930360 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:39.929830    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="289a6c0a-140b-471a-a549-cf3f2d07dabc" containerName="revoke-keys"
Oct 29 22:53:40.129943 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.129868    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:40.141949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.141916    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.172706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.172674    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"]
Oct 29 22:53:40.210880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.210837    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-socket\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.211081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.210935    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-envoy\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.211081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.211036    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istiod-ca-cert\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.211265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.211230    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-credential-socket\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.211588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.211568    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-token\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.211878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.211855    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-data\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.211935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.211909    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-podinfo\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.212023 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.212005    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84c8v\" (UniqueName: \"kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-kube-api-access-84c8v\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.212093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.212031    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-certs\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.240555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.240535    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7be71d-openshift-default-dockercfg-9tgt6\""
Oct 29 22:53:40.313382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.313315    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-credential-socket\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.313541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.313497    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-token\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.313668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.313652    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-data\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.313716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.313680    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-podinfo\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.313774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.313704    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-credential-socket\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.313774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.313733    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84c8v\" (UniqueName: \"kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-kube-api-access-84c8v\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.313774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.313761    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-certs\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.313951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.313797    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-socket\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.313951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.313833    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-envoy\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.313951 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.313868    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istiod-ca-cert\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.314118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.314000    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-data\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.314118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.314063    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-certs\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.314218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.314156    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-socket\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.314547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.314531    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istiod-ca-cert\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.316099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.316078    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-envoy\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.316342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.316306    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-podinfo\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.334396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.334368    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-token\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.334518 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.334509    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84c8v\" (UniqueName: \"kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-kube-api-access-84c8v\") pod \"e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") " pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.475250 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.475174    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:40.514852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.514821    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:40.844688 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:53:40.844657    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f45ec0f_a12c_4de7_a655_51ba0d0cd85f.slice/crio-ce58061bad85e23ce0e72ee8229af99f9f7e35f52e2b63bd74f2fe930dcfcc49 WatchSource:0}: Error finding container ce58061bad85e23ce0e72ee8229af99f9f7e35f52e2b63bd74f2fe930dcfcc49: Status 404 returned error can't find the container with id ce58061bad85e23ce0e72ee8229af99f9f7e35f52e2b63bd74f2fe930dcfcc49
Oct 29 22:53:40.847319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.847292    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"]
Oct 29 22:53:40.847657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.847622    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:53:40.847730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.847714    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:53:40.847768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.847752    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:53:40.942636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.942600    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"]
Oct 29 22:53:40.947880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.947856    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:40.973869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.973831    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"]
Oct 29 22:53:40.993392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:40.992269    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-920032-openshift-default-dockercfg-72mrh\""
Oct 29 22:53:41.028347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.027805    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad192f3-f04a-439e-8b84-aeb905d0813c-istiod-ca-cert\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.028347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.027859    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-token\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.028347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.027926    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-envoy\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.028347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.027961    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.028347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.028063    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-credential-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.028347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.028170    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-podinfo\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.028347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.028261    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtxdh\" (UniqueName: \"kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-kube-api-access-xtxdh\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.028347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.028309    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-certs\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.028995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.028394    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-data\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.130393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.130270    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad192f3-f04a-439e-8b84-aeb905d0813c-istiod-ca-cert\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.130393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.130341    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-token\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.130393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.130380    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-envoy\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.130654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.130417    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.130654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.130518    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-credential-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.130654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.130558    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-podinfo\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.130654 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.130609    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtxdh\" (UniqueName: \"kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-kube-api-access-xtxdh\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.131171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.131128    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-certs\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.131171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.131152    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.131171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.131166    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-credential-socket\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.131449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.131154    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad192f3-f04a-439e-8b84-aeb905d0813c-istiod-ca-cert\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.131449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.131267    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-data\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.131579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.131494    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-certs\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.131673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.131649    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-data\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.133535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.133512    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-envoy\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.134122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.134094    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-podinfo\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.152362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.152337    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtxdh\" (UniqueName: \"kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-kube-api-access-xtxdh\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.160268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.159711    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-token\") pod \"e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") " pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.245536 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.245499    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:41.266718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.266679    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:41.644563 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:53:41.644520    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad192f3_f04a_439e_8b84_aeb905d0813c.slice/crio-063372a11f41de02e5fdd7aca8898d24303b7ebf7ab8f5d220f4144e63913f3f WatchSource:0}: Error finding container 063372a11f41de02e5fdd7aca8898d24303b7ebf7ab8f5d220f4144e63913f3f: Status 404 returned error can't find the container with id 063372a11f41de02e5fdd7aca8898d24303b7ebf7ab8f5d220f4144e63913f3f
Oct 29 22:53:41.647759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.647716    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:53:41.647889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.647795    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:53:41.647889 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.647839    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:53:41.648304 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.648277    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"]
Oct 29 22:53:41.713099 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.713069    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4" event={"ID":"4ad192f3-f04a-439e-8b84-aeb905d0813c","Type":"ContainerStarted","Data":"063372a11f41de02e5fdd7aca8898d24303b7ebf7ab8f5d220f4144e63913f3f"}
Oct 29 22:53:41.714999 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.714972    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" event={"ID":"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f","Type":"ContainerStarted","Data":"1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0"}
Oct 29 22:53:41.715112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.715019    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" event={"ID":"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f","Type":"ContainerStarted","Data":"ce58061bad85e23ce0e72ee8229af99f9f7e35f52e2b63bd74f2fe930dcfcc49"}
Oct 29 22:53:41.769014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:41.768949    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" podStartSLOduration=2.768929882 podStartE2EDuration="2.768929882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:41.761826044 +0000 UTC m=+2363.786602206" watchObservedRunningTime="2026-10-29 22:53:41.768929882 +0000 UTC m=+2363.793706014"
Oct 29 22:53:42.489057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:42.488969    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:42.694349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:42.692774    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:42.780830 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:42.780739    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4" event={"ID":"4ad192f3-f04a-439e-8b84-aeb905d0813c","Type":"ContainerStarted","Data":"81885ba6b8ab1e412b3d3916452c395f7f47abe48d2383c488c996b0aa51bf7e"}
Oct 29 22:53:42.785438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:42.785409    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:42.791665 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:42.791630    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:53:42.882317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:42.882254    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4" podStartSLOduration=2.882235955 podStartE2EDuration="2.882235955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:42.880666123 +0000 UTC m=+2364.905442280" watchObservedRunningTime="2026-10-29 22:53:42.882235955 +0000 UTC m=+2364.907012087"
Oct 29 22:53:43.276353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.254895    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:43.290541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.290506    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:43.413353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.400904    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 29 22:53:43.413353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.400987    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4" podUID="4ad192f3-f04a-439e-8b84-aeb905d0813c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 29 22:53:43.512779 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.512731    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:43.582001 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.581891    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:43.588392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.588353    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:43.614394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.609466    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:43.662455 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.662405    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:43.682139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.682096    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:43.698348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.696405    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:43.821179 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.821099    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main" probeResult="failure" output="Get \"https://10.132.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:43.854449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.841499    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:43.880354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.880063    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:43.948384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.947415    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:43.948384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.947806    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:53:43.974350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:43.973548    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:53:44.019179 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:44.019144    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:44.278351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:44.275714    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 29 22:53:44.278351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:44.275790    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4" podUID="4ad192f3-f04a-439e-8b84-aeb905d0813c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 29 22:53:44.408413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:44.408374    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:44.539869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:44.539795    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:45.093636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:45.093388    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:45.273348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:45.273127    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 29 22:53:45.273348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:45.273184    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4" podUID="4ad192f3-f04a-439e-8b84-aeb905d0813c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 29 22:53:46.210218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:46.210176    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:46.291346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:46.290481    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 29 22:53:46.291346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:46.290539    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4" podUID="4ad192f3-f04a-439e-8b84-aeb905d0813c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 29 22:53:46.339459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:46.339421    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:47.132979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:47.132934    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:47.267743 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:47.267708    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 29 22:53:47.268319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:47.268282    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4" podUID="4ad192f3-f04a-439e-8b84-aeb905d0813c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 29 22:53:48.031364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:48.031289    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"]
Oct 29 22:53:48.045579 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:48.045523    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e6d451d-8frgz"]
Oct 29 22:53:48.060827 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:53:48.060791    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:53:48.113093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:48.113058    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67418cf-95b9-44fd-bbef-aa3052ee6e51" path="/var/lib/kubelet/pods/f67418cf-95b9-44fd-bbef-aa3052ee6e51/volumes"
Oct 29 22:53:48.267109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:48.267077    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 29 22:53:48.267281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:48.267132    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4" podUID="4ad192f3-f04a-439e-8b84-aeb905d0813c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 29 22:53:48.854824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:48.854793    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:49.438501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:49.438468    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:49.439035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:49.438994    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:49.439959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:49.439935    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 22:53:50.494230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:50.494199    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:52.256071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:52.256035    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:54.048834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:54.048799    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:55.287249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:55.287212    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:55.463463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:55.463404    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:56.462781 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:56.462743    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:58.191581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:58.191547    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:59.843411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:53:59.843302    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:00.584873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:00.584834    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:01.704647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:01.704609    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:02.923043 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:02.923012    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:03.669897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:03.669865    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:05.473575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:05.473536    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:06.667542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:06.667510    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:07.354181 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:07.354152    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:07.752232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:07.752198    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:09.124238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:09.124198    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:10.813674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:10.813630    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:11.284223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.284167    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:11.821622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.821590    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"]
Oct 29 22:54:11.828308 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.828283    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:11.850810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.850772    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"]
Oct 29 22:54:11.851678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.851306    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8c4b02-openshift-default-dockercfg-q28dz\""
Oct 29 22:54:11.923938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.923901    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5299767-caa1-4246-b4c1-9217642edfab-istio-podinfo\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:11.924171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.923950    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-socket\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:11.924171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.924081    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-envoy\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:11.924171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.924139    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-data\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:11.924386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.924245    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-certs\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:11.924458 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.924439    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5299767-caa1-4246-b4c1-9217642edfab-istiod-ca-cert\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:11.924647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.924630    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft6fp\" (UniqueName: \"kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-kube-api-access-ft6fp\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:11.924693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.924659    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-istio-token\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:11.924811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:11.924790    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-credential-socket\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.026531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.026484    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft6fp\" (UniqueName: \"kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-kube-api-access-ft6fp\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.026735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.026576    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-istio-token\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.026735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.026706    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-credential-socket\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.026893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.026835    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5299767-caa1-4246-b4c1-9217642edfab-istio-podinfo\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.026959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.026872    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-socket\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.027025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.026966    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-envoy\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.027085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.027044    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-data\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.027146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.027116    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-certs\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.027285 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.027267    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5299767-caa1-4246-b4c1-9217642edfab-istiod-ca-cert\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.027991 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.027966    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5299767-caa1-4246-b4c1-9217642edfab-istiod-ca-cert\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.028746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.028726    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-credential-socket\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.031499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.031430    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5299767-caa1-4246-b4c1-9217642edfab-istio-podinfo\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.032473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.031828    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-data\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.032473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.032149    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-certs\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.032473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.032413    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-socket\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.034888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.034857    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-envoy\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.076178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.076102    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-istio-token\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.085412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.077029    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft6fp\" (UniqueName: \"kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-kube-api-access-ft6fp\") pod \"e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") " pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.229436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.229406    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:12.604061 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:54:12.604027    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5299767_caa1_4246_b4c1_9217642edfab.slice/crio-0e7d7aaf73e88403d0e4788c89f07990f4d5d9534fd395b011ad939fe634f7ff WatchSource:0}: Error finding container 0e7d7aaf73e88403d0e4788c89f07990f4d5d9534fd395b011ad939fe634f7ff: Status 404 returned error can't find the container with id 0e7d7aaf73e88403d0e4788c89f07990f4d5d9534fd395b011ad939fe634f7ff
Oct 29 22:54:12.606892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.606856    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"]
Oct 29 22:54:12.607744 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.607702    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:54:12.607848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.607793    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:54:12.607848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:12.607840    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:54:13.104834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:13.104798    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:13.215238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:13.215204    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" event={"ID":"d5299767-caa1-4246-b4c1-9217642edfab","Type":"ContainerStarted","Data":"992f3625e72cc437f6d1d36a41ca8cada4b50ee622d49920f870b1d85fb341ba"}
Oct 29 22:54:13.215441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:13.215283    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" event={"ID":"d5299767-caa1-4246-b4c1-9217642edfab","Type":"ContainerStarted","Data":"0e7d7aaf73e88403d0e4788c89f07990f4d5d9534fd395b011ad939fe634f7ff"}
Oct 29 22:54:13.295959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:13.295909    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" podStartSLOduration=2.295895138 podStartE2EDuration="2.295895138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:13.294387222 +0000 UTC m=+2395.319163354" watchObservedRunningTime="2026-10-29 22:54:13.295895138 +0000 UTC m=+2395.320671269"
Oct 29 22:54:13.735348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:13.732371    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:14.175404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:14.172869    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:14.227348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:14.198099    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:14.357466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:14.357067    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:14.616203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:14.616089    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:15.056350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:15.047569    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:15.294348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:15.293913    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:15.295908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:15.295477    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:54:15.886921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:15.886891    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:16.026582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:16.026547    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:16.175168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:16.175082    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:17.284786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:17.284749    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:17.713345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:17.713275    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:17.837188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:17.837153    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:17.859074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:17.859044    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"]
Oct 29 22:54:17.863778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:17.863762    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"
Oct 29 22:54:17.883954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:17.883929    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"]
Oct 29 22:54:17.890950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:17.890915    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8c4b02\""
Oct 29 22:54:17.941749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:17.941607    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8\" (UID: \"2e4c6741-fe7a-4410-9a1b-a47d2021bc58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"
Oct 29 22:54:17.942236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:17.942211    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92zjp\" (UniqueName: \"kubernetes.io/projected/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-kube-api-access-92zjp\") pod \"payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8\" (UID: \"2e4c6741-fe7a-4410-9a1b-a47d2021bc58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"
Oct 29 22:54:18.060408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.055894    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"]
Oct 29 22:54:18.120348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.114916    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92zjp\" (UniqueName: \"kubernetes.io/projected/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-kube-api-access-92zjp\") pod \"payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8\" (UID: \"2e4c6741-fe7a-4410-9a1b-a47d2021bc58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"
Oct 29 22:54:18.120348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.115231    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8\" (UID: \"2e4c6741-fe7a-4410-9a1b-a47d2021bc58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"
Oct 29 22:54:18.120348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.116183    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8\" (UID: \"2e4c6741-fe7a-4410-9a1b-a47d2021bc58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"
Oct 29 22:54:18.143255 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.140776    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8c4b02-dockercfg-w9jl4\""
Oct 29 22:54:18.172816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.172786    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"]
Oct 29 22:54:18.174409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.174109    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"
Oct 29 22:54:18.202201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.202164    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92zjp\" (UniqueName: \"kubernetes.io/projected/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-kube-api-access-92zjp\") pod \"payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8\" (UID: \"2e4c6741-fe7a-4410-9a1b-a47d2021bc58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"
Oct 29 22:54:18.238378 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:54:18.238347    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:54:18.328392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.328257    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058881c3-8417-4e47-b394-0d29112080a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj\" (UID: \"058881c3-8417-4e47-b394-0d29112080a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"
Oct 29 22:54:18.328884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.328591    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pmgj\" (UniqueName: \"kubernetes.io/projected/058881c3-8417-4e47-b394-0d29112080a5-kube-api-access-4pmgj\") pod \"payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj\" (UID: \"058881c3-8417-4e47-b394-0d29112080a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"
Oct 29 22:54:18.430192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.430155    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pmgj\" (UniqueName: \"kubernetes.io/projected/058881c3-8417-4e47-b394-0d29112080a5-kube-api-access-4pmgj\") pod \"payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj\" (UID: \"058881c3-8417-4e47-b394-0d29112080a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"
Oct 29 22:54:18.430397 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.430288    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058881c3-8417-4e47-b394-0d29112080a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj\" (UID: \"058881c3-8417-4e47-b394-0d29112080a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"
Oct 29 22:54:18.430988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.430970    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058881c3-8417-4e47-b394-0d29112080a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj\" (UID: \"058881c3-8417-4e47-b394-0d29112080a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"
Oct 29 22:54:18.450413 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.450369    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pmgj\" (UniqueName: \"kubernetes.io/projected/058881c3-8417-4e47-b394-0d29112080a5-kube-api-access-4pmgj\") pod \"payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj\" (UID: \"058881c3-8417-4e47-b394-0d29112080a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"
Oct 29 22:54:18.511474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.511450    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8c4b02-dockercfg-w9jl4\""
Oct 29 22:54:18.515881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.515854    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"
Oct 29 22:54:18.517033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.517017    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"
Oct 29 22:54:18.645618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.645600    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:54:18.680109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.680022    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:54:18.767998 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.767960    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"]
Oct 29 22:54:18.773909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.773885    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:18.795602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.795575    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"]
Oct 29 22:54:18.800151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.800128    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-eb02df-openshift-default-dockercfg-8bq6z\""
Oct 29 22:54:18.832928 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:54:18.832876    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058881c3_8417_4e47_b394_0d29112080a5.slice/crio-7091e2d6ef330c5203dc4261448ba787fc75ee8915132fc277bed6b54fb59080 WatchSource:0}: Error finding container 7091e2d6ef330c5203dc4261448ba787fc75ee8915132fc277bed6b54fb59080: Status 404 returned error can't find the container with id 7091e2d6ef330c5203dc4261448ba787fc75ee8915132fc277bed6b54fb59080
Oct 29 22:54:18.837366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.837342    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"]
Oct 29 22:54:18.865664 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.865637    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"]
Oct 29 22:54:18.937411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.937388    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-data\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:18.937545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.937450    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-certs\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:18.937619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.937541    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:18.937619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.937570    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-podinfo\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:18.937747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.937692    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-credential-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:18.937802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.937749    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54sb9\" (UniqueName: \"kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-kube-api-access-54sb9\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:18.937869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.937849    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-token\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:18.937975 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.937960    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-envoy\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:18.938022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:18.937986    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istiod-ca-cert\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.039526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.039493    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-certs\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.039693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.039543    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.039693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.039656    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-podinfo\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.039822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.039760    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-credential-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.039883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.039826    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54sb9\" (UniqueName: \"kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-kube-api-access-54sb9\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.039976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.039902    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-token\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.039976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.039924    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-certs\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.039976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.039950    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.040156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.040052    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-envoy\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.040156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.040080    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istiod-ca-cert\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.040156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.040092    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-credential-socket\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.040293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.040274    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-data\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.040612 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.040589    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-data\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.040748 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.040732    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istiod-ca-cert\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.042287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.042264    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-envoy\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.042391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.042285    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-podinfo\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.060582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.060558    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:19.060823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.060799    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-token\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.061026 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.061008    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54sb9\" (UniqueName: \"kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-kube-api-access-54sb9\") pod \"e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") " pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.090760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.090442    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:19.294305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.294273    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"]
Oct 29 22:54:19.301682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.301652    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:19.317265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.317234    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"]
Oct 29 22:54:19.332374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.323829    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:54:19.332374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.323904    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:54:19.334071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.334038    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj" event={"ID":"058881c3-8417-4e47-b394-0d29112080a5","Type":"ContainerStarted","Data":"45245e311a5f8f73e004ba13cce014569af286ca40e8d64a7abda95a7433be9b"}
Oct 29 22:54:19.334199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.334091    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj" event={"ID":"058881c3-8417-4e47-b394-0d29112080a5","Type":"ContainerStarted","Data":"7091e2d6ef330c5203dc4261448ba787fc75ee8915132fc277bed6b54fb59080"}
Oct 29 22:54:19.334199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.334133    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"
Oct 29 22:54:19.336379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.336350    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8" event={"ID":"2e4c6741-fe7a-4410-9a1b-a47d2021bc58","Type":"ContainerStarted","Data":"5085faa350710b341f15349ae2693b58b4873c6efaebe397233daf8f0630cff1"}
Oct 29 22:54:19.336483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.336430    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8" event={"ID":"2e4c6741-fe7a-4410-9a1b-a47d2021bc58","Type":"ContainerStarted","Data":"861a0bca83735d3cfd5bbc436f1c1a1472a3470708ef4e8a5b0069a1f8677349"}
Oct 29 22:54:19.336544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.336490    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"
Oct 29 22:54:19.391496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.391418    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj" podStartSLOduration=2.391406368 podStartE2EDuration="2.391406368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:19.389602107 +0000 UTC m=+2401.414378249" watchObservedRunningTime="2026-10-29 22:54:19.391406368 +0000 UTC m=+2401.416182511"
Oct 29 22:54:19.437108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.437066    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8" podStartSLOduration=2.437053788 podStartE2EDuration="2.437053788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:19.43220512 +0000 UTC m=+2401.456981251" watchObservedRunningTime="2026-10-29 22:54:19.437053788 +0000 UTC m=+2401.461829932"
Oct 29 22:54:19.448495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.448457    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c420727-65565\" (UID: \"3192b217-2d8a-4c01-b959-39d74e31b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:19.448636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.448620    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bck\" (UniqueName: \"kubernetes.io/projected/3192b217-2d8a-4c01-b959-39d74e31b70a-kube-api-access-q8bck\") pod \"maas-api-revoke-sub-1c420727-65565\" (UID: \"3192b217-2d8a-4c01-b959-39d74e31b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:19.448849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.448822    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c420727-65565\" (UID: \"3192b217-2d8a-4c01-b959-39d74e31b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:19.450616 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:54:19.450587    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fff8e8_73e3_4a52_8f3b_032ea9abdfcc.slice/crio-4a743a5b408685fded77319dbafbdf898ae555b108aed0e41d1119de60f05f99 WatchSource:0}: Error finding container 4a743a5b408685fded77319dbafbdf898ae555b108aed0e41d1119de60f05f99: Status 404 returned error can't find the container with id 4a743a5b408685fded77319dbafbdf898ae555b108aed0e41d1119de60f05f99
Oct 29 22:54:19.453655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.453622    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:54:19.453814 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.453703    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:54:19.453814 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.453740    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"]
Oct 29 22:54:19.453814 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.453754    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:54:19.544883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.544849    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:19.551137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.551103    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c420727-65565\" (UID: \"3192b217-2d8a-4c01-b959-39d74e31b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:19.551265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.551249    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c420727-65565\" (UID: \"3192b217-2d8a-4c01-b959-39d74e31b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:19.551493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.551466    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bck\" (UniqueName: \"kubernetes.io/projected/3192b217-2d8a-4c01-b959-39d74e31b70a-kube-api-access-q8bck\") pod \"maas-api-revoke-sub-1c420727-65565\" (UID: \"3192b217-2d8a-4c01-b959-39d74e31b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:19.552098 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.552073    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c420727-65565\" (UID: \"3192b217-2d8a-4c01-b959-39d74e31b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:19.553630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.553603    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c420727-65565\" (UID: \"3192b217-2d8a-4c01-b959-39d74e31b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:19.573594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.573569    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bck\" (UniqueName: \"kubernetes.io/projected/3192b217-2d8a-4c01-b959-39d74e31b70a-kube-api-access-q8bck\") pod \"maas-api-revoke-sub-1c420727-65565\" (UID: \"3192b217-2d8a-4c01-b959-39d74e31b70a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:19.625737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.625709    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:19.658729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:19.658699    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:20.038459 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:54:20.038410    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3192b217_2d8a_4c01_b959_39d74e31b70a.slice/crio-0534091c417f47023c6d09cf6d7266899367c142fc904d99577be1b7663dc313 WatchSource:0}: Error finding container 0534091c417f47023c6d09cf6d7266899367c142fc904d99577be1b7663dc313: Status 404 returned error can't find the container with id 0534091c417f47023c6d09cf6d7266899367c142fc904d99577be1b7663dc313
Oct 29 22:54:20.048614 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:20.048571    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"]
Oct 29 22:54:20.349359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:20.349251    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5" event={"ID":"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc","Type":"ContainerStarted","Data":"ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93"}
Oct 29 22:54:20.349359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:20.349309    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5" event={"ID":"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc","Type":"ContainerStarted","Data":"4a743a5b408685fded77319dbafbdf898ae555b108aed0e41d1119de60f05f99"}
Oct 29 22:54:20.350786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:20.350761    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565" event={"ID":"3192b217-2d8a-4c01-b959-39d74e31b70a","Type":"ContainerStarted","Data":"0534091c417f47023c6d09cf6d7266899367c142fc904d99577be1b7663dc313"}
Oct 29 22:54:20.411096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:20.411052    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5" podStartSLOduration=2.411038898 podStartE2EDuration="2.411038898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:20.406281545 +0000 UTC m=+2402.431057688" watchObservedRunningTime="2026-10-29 22:54:20.411038898 +0000 UTC m=+2402.435815029"
Oct 29 22:54:20.828575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:20.828535    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:21.096600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:21.092584    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:21.096815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:21.096757    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 29 22:54:21.096894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:21.096810    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5" podUID="27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 29 22:54:21.362548 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:21.362483    4860 generic.go:356] "Generic (PLEG): container finished" podID="3192b217-2d8a-4c01-b959-39d74e31b70a" containerID="98949dbac696c2dfd25919aca39877b9292195525ea6ca35729987c95dc0d02e" exitCode=0
Oct 29 22:54:21.362871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:21.362571    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565" event={"ID":"3192b217-2d8a-4c01-b959-39d74e31b70a","Type":"ContainerDied","Data":"98949dbac696c2dfd25919aca39877b9292195525ea6ca35729987c95dc0d02e"}
Oct 29 22:54:22.092709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.092664    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 29 22:54:22.092931 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.092734    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5" podUID="27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 29 22:54:22.606735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.606708    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:22.657337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.657263    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-openshift-ca\") pod \"3192b217-2d8a-4c01-b959-39d74e31b70a\" (UID: \"3192b217-2d8a-4c01-b959-39d74e31b70a\") "
Oct 29 22:54:22.657482 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.657414    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3192b217-2d8a-4c01-b959-39d74e31b70a-kube-api-access-q8bck\" (UniqueName: \"kubernetes.io/projected/3192b217-2d8a-4c01-b959-39d74e31b70a-kube-api-access-q8bck\") pod \"3192b217-2d8a-4c01-b959-39d74e31b70a\" (UID: \"3192b217-2d8a-4c01-b959-39d74e31b70a\") "
Oct 29 22:54:22.657582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.657500    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-xks-ca\") pod \"3192b217-2d8a-4c01-b959-39d74e31b70a\" (UID: \"3192b217-2d8a-4c01-b959-39d74e31b70a\") "
Oct 29 22:54:22.657630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.657577    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-openshift-ca" pod "3192b217-2d8a-4c01-b959-39d74e31b70a" (UID: "3192b217-2d8a-4c01-b959-39d74e31b70a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:22.658264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.658242    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:54:22.659629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.659602    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-xks-ca" pod "3192b217-2d8a-4c01-b959-39d74e31b70a" (UID: "3192b217-2d8a-4c01-b959-39d74e31b70a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:22.659873 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.659853    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3192b217-2d8a-4c01-b959-39d74e31b70a-kube-api-access-q8bck" pod "3192b217-2d8a-4c01-b959-39d74e31b70a" (UID: "3192b217-2d8a-4c01-b959-39d74e31b70a"). InnerVolumeSpecName "kube-api-access-q8bck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:22.717356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.715305    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:22.761543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.761444    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3192b217-2d8a-4c01-b959-39d74e31b70a-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:54:22.761543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:22.761481    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8bck\" (UniqueName: \"kubernetes.io/projected/3192b217-2d8a-4c01-b959-39d74e31b70a-kube-api-access-q8bck\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:54:23.094266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:23.093657    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 29 22:54:23.094266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:23.093718    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5" podUID="27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 29 22:54:23.407850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:23.407763    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"
Oct 29 22:54:23.407850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:23.407792    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565" event={"ID":"3192b217-2d8a-4c01-b959-39d74e31b70a","Type":"ContainerDied","Data":"0534091c417f47023c6d09cf6d7266899367c142fc904d99577be1b7663dc313"}
Oct 29 22:54:23.407850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:23.407841    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0534091c417f47023c6d09cf6d7266899367c142fc904d99577be1b7663dc313"
Oct 29 22:54:23.431336 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:23.431302    4860 scope.go:122] "RemoveContainer" containerID="bda4345e104cc50b8d72d9a3caec01feb72fd764973d395c5ed6dceba8845162"
Oct 29 22:54:23.442969 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:23.442951    4860 scope.go:122] "RemoveContainer" containerID="ae294e857339076e12e4e262088cd0af5612264dc593031f4a8e0aefd5dbbe0b"
Oct 29 22:54:23.453188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:23.453162    4860 scope.go:122] "RemoveContainer" containerID="c8f0f363e3a65b7b4c86079b84c438e86282ac1556be007adf4a18c36c3d4ea7"
Oct 29 22:54:23.463273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:23.463243    4860 scope.go:122] "RemoveContainer" containerID="b5bbd62bfc29511df8e8bab1a1c1f773a080a0870136fabf896ac15dc175ce79"
Oct 29 22:54:23.499055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:23.499032    4860 scope.go:122] "RemoveContainer" containerID="ee1195a47a4094b0d5ba253c8c0bb0c91afdb210d35a589620b59260c37f091b"
Oct 29 22:54:23.509176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:23.509159    4860 scope.go:122] "RemoveContainer" containerID="67da4a43a901d432493cadd867950ef7b888d8b745c54521cf8c1fdd31464992"
Oct 29 22:54:23.518953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:23.518776    4860 scope.go:122] "RemoveContainer" containerID="a7e0684d26a9dbc9a20c4400d57a443e7fe634edff56386dd2b80fd45b9c9adf"
Oct 29 22:54:24.092642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:24.092606    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 29 22:54:24.093084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:24.092665    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5" podUID="27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 29 22:54:24.406806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:24.406722    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:25.092655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:25.092616    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 29 22:54:25.093265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:25.093196    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5" podUID="27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 29 22:54:26.091310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:26.091272    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body=
Oct 29 22:54:26.091310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:26.091336    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5" podUID="27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused"
Oct 29 22:54:26.230898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:26.230850    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:26.375863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:26.375768    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:26.561595 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:26.561557    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:27.096604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:27.096573    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:27.097064 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:27.097044    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:27.097823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:27.097806    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 22:54:28.073642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:28.073607    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:29.627123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.627090    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"]
Oct 29 22:54:29.628041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.628017    4860 cpu_manager.go:414] "containerMap: removing container" podUID="3192b217-2d8a-4c01-b959-39d74e31b70a" containerName="revoke-keys"
Oct 29 22:54:29.628041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.628041    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3192b217-2d8a-4c01-b959-39d74e31b70a" containerName="revoke-keys"
Oct 29 22:54:29.628208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.628198    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="3192b217-2d8a-4c01-b959-39d74e31b70a" containerName="revoke-keys"
Oct 29 22:54:29.633760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.633738    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:29.650283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.650258    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"]
Oct 29 22:54:29.657341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.657302    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:54:29.657512 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.657403    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:54:29.711763 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.711731    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2491765-zl4j8\" (UID: \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:29.711982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.711965    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkpmn\" (UniqueName: \"kubernetes.io/projected/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-kube-api-access-kkpmn\") pod \"maas-api-revoke-sub-c2491765-zl4j8\" (UID: \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:29.712036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.712008    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2491765-zl4j8\" (UID: \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:29.755939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.755911    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:29.813219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.813185    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkpmn\" (UniqueName: \"kubernetes.io/projected/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-kube-api-access-kkpmn\") pod \"maas-api-revoke-sub-c2491765-zl4j8\" (UID: \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:29.813546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.813244    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2491765-zl4j8\" (UID: \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:29.813546 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.813347    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2491765-zl4j8\" (UID: \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:29.814115 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.814088    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2491765-zl4j8\" (UID: \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:29.816082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.816052    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2491765-zl4j8\" (UID: \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:29.838075 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.834908    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkpmn\" (UniqueName: \"kubernetes.io/projected/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-kube-api-access-kkpmn\") pod \"maas-api-revoke-sub-c2491765-zl4j8\" (UID: \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:29.949457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:29.949427    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:30.254986 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:54:30.254957    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2333cdd4_c0fe_4ba2_822e_3d55c11ca1cf.slice/crio-eae41b1342142463e782331fdf7858fca4e2af3bc7c11040a86aceceb85a142e WatchSource:0}: Error finding container eae41b1342142463e782331fdf7858fca4e2af3bc7c11040a86aceceb85a142e: Status 404 returned error can't find the container with id eae41b1342142463e782331fdf7858fca4e2af3bc7c11040a86aceceb85a142e
Oct 29 22:54:30.260593 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:30.260565    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"]
Oct 29 22:54:30.354213 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:30.354186    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"
Oct 29 22:54:30.354319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:30.354253    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"
Oct 29 22:54:30.514046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:30.513964    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8" event={"ID":"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf","Type":"ContainerStarted","Data":"eae41b1342142463e782331fdf7858fca4e2af3bc7c11040a86aceceb85a142e"}
Oct 29 22:54:31.524914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:31.524825    4860 generic.go:356] "Generic (PLEG): container finished" podID="2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf" containerID="b6fe6971cdd40d36fb072575c04363302f4729a4eda70c0df00c18aa4fec1f8a" exitCode=0
Oct 29 22:54:31.525399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:31.524910    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8" event={"ID":"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf","Type":"ContainerDied","Data":"b6fe6971cdd40d36fb072575c04363302f4729a4eda70c0df00c18aa4fec1f8a"}
Oct 29 22:54:31.783283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:31.783178    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:32.711139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:32.711111    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:32.764879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:32.764849    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-kube-api-access-kkpmn\" (UniqueName: \"kubernetes.io/projected/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-kube-api-access-kkpmn\") pod \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\" (UID: \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\") "
Oct 29 22:54:32.765094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:32.764907    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-openshift-ca\") pod \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\" (UID: \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\") "
Oct 29 22:54:32.765174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:32.765091    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-xks-ca\") pod \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\" (UID: \"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf\") "
Oct 29 22:54:32.766090 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:32.766056    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-openshift-ca" pod "2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf" (UID: "2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:32.766218 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:32.766196    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:54:32.767828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:32.767805    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-kube-api-access-kkpmn" pod "2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf" (UID: "2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf"). InnerVolumeSpecName "kube-api-access-kkpmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:32.768650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:32.768491    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-xks-ca" pod "2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf" (UID: "2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:32.876687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:32.876258    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkpmn\" (UniqueName: \"kubernetes.io/projected/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-kube-api-access-kkpmn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:54:32.876687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:32.876293    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:54:33.458635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:33.458606    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:33.545583 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:33.545558    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"
Oct 29 22:54:33.545767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:33.545554    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8" event={"ID":"2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf","Type":"ContainerDied","Data":"eae41b1342142463e782331fdf7858fca4e2af3bc7c11040a86aceceb85a142e"}
Oct 29 22:54:33.545767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:33.545687    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eae41b1342142463e782331fdf7858fca4e2af3bc7c11040a86aceceb85a142e"
Oct 29 22:54:35.181487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:35.181450    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:36.947867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:36.947829    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:38.673865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:38.673822    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:40.846141 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:40.846103    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:42.566344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:42.566276    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:43.634215 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:43.634173    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:44.277288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:44.277238    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:45.407224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:45.407188    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:46.032635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:46.032597    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:47.932531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:47.932496    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:48.065420 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:54:48.065376    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:54:49.546531 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:49.546497    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:51.168898 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:51.168865    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:52.917875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:52.917842    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:54.257260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:54.257228    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:54.367231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:54.367199    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:54.562480 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:54.562409    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:56.241053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:56.241015    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:58.074954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:58.074912    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:58.191355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:58.187113    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:59.839857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:59.839825    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:59.959411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:54:59.959379    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:00.214470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:00.214436    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:01.651214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:01.651184    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:03.575338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:03.575282    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:05.404004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:05.403917    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:05.570910 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:05.568619    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:55:05.609357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:05.606670    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:05.715447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:05.681001    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:05.715447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:05.695143    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:05.715447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:05.699583    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:05.821892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:05.821821    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:05.823659 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:05.823620    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main" probeResult="failure" output="Get \"https://10.132.0.154:8000/ready\": context deadline exceeded"
Oct 29 22:55:05.950543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:05.950488    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:05.973741 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:05.973634    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:55:05.975877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:05.975840    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Oct 29 22:55:06.165350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.063029    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m" podUID="da28494b-2ff0-4782-b53f-2583dc161b9d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:06.340409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.336521    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:06.340409 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.337000    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:55:06.609033 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.608917    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:06.626114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.611950    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:06.626114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.612000    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:55:06.626114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.617418    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:55:06.626114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.617478    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:55:06.652775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.652418    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:06.712065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.712021    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:06.712712 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.712688    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:55:06.715101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.715072    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:06.732753 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.732728    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:55:06.764955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.764923    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:55:06.784373 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.766193    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:55:06.787563 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.787534    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:55:06.820160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:06.818839    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:55:07.114386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:07.114319    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:08.965199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:08.965167    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:10.612645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:10.612605    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:12.331510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:12.331467    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:13.838346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:13.837446    4860 patch_prober.go:29] interesting pod/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:55:13.838346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:13.837557    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:14.240251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:14.240218    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:14.366375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:14.366311    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:15.516416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:15.516346    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:16.227461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:16.227428    4860 generic.go:356] "Generic (PLEG): container finished" podID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerID="1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0" exitCode=0
Oct 29 22:55:16.227644 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:16.227497    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" event={"ID":"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f","Type":"ContainerDied","Data":"1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0"}
Oct 29 22:55:16.227891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:16.227870    4860 scope.go:122] "RemoveContainer" containerID="1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0"
Oct 29 22:55:16.229624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:16.229594    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:55:16.229706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:16.229671    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:55:16.229748 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:16.229705    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:55:16.267037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:16.267015    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:16.809204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:16.809166    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:17.089278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:17.089199    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:17.214358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:17.213741    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:17.253347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:17.249172    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" event={"ID":"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f","Type":"ContainerStarted","Data":"28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173"}
Oct 29 22:55:17.485347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:17.482756    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:55:17.485347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:17.482900    4860 patch_prober.go:29] interesting pod/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 29 22:55:17.485347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:17.482961    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 29 22:55:18.060395 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:55:18.060362    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:55:18.099978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:18.099943    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:18.478729 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:18.478694    4860 patch_prober.go:29] interesting pod/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 29 22:55:18.479020 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:18.478988    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 29 22:55:19.476350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:19.475654    4860 patch_prober.go:29] interesting pod/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 29 22:55:19.476350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:19.475726    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 29 22:55:19.754289 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:19.754198    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:20.486746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:20.486688    4860 patch_prober.go:29] interesting pod/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 29 22:55:20.487400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:20.487366    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 29 22:55:21.487710 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:21.487673    4860 patch_prober.go:29] interesting pod/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 29 22:55:21.488311 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:21.488279    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 29 22:55:21.550184 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:21.550159    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:22.493071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:22.493031    4860 patch_prober.go:29] interesting pod/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body=
Oct 29 22:55:22.493615 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:22.493091    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused"
Oct 29 22:55:23.265597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.265563    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:23.480446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.480414    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:55:23.480926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.480900    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:55:23.481657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.481639    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 22:55:23.620318 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.620222    4860 scope.go:122] "RemoveContainer" containerID="248f0ac7830832180fd0704ef77230bdc7a3c64fa6c6e3195d5583ff6874c4db"
Oct 29 22:55:23.636131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.636104    4860 scope.go:122] "RemoveContainer" containerID="c970869790f420de57166e1cda1055a04123537c10fb0cf31b64d9eaad59929f"
Oct 29 22:55:23.647944 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.647923    4860 scope.go:122] "RemoveContainer" containerID="b3bb6db2065d2f80140c0a62338e609337c6c69e694d025651a14fc749fed2af"
Oct 29 22:55:23.661560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.661532    4860 scope.go:122] "RemoveContainer" containerID="70262766aa27b24d779b05ef9c427579f7dc0a1e0248735bcd81c8b9c0078ff0"
Oct 29 22:55:23.671888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.671866    4860 scope.go:122] "RemoveContainer" containerID="c31b728797345e08402d9b25b8926818d530d8328878e689b413e569e3b6d854"
Oct 29 22:55:23.681904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.681879    4860 scope.go:122] "RemoveContainer" containerID="f8d10ecbbd23019d2a287e4f2b31299805a6342d495c6a6b1eb04842ce93d2f0"
Oct 29 22:55:23.706668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.706643    4860 scope.go:122] "RemoveContainer" containerID="4be32425b556c4a0ef733b12f03726db61e10bc531e5b70b3c62e604dd9ccd07"
Oct 29 22:55:23.726030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.726006    4860 scope.go:122] "RemoveContainer" containerID="ed64529fc35530baccaab812c57f5b581a85fe548892b848be8daea2e7fd6602"
Oct 29 22:55:23.736857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.736838    4860 scope.go:122] "RemoveContainer" containerID="7a055cf03557ac8fce95608a81532977488cf68a2763b8e2e0504d11140ea83f"
Oct 29 22:55:23.747592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:23.747561    4860 scope.go:122] "RemoveContainer" containerID="a4f74136f0154a6044c71d945abe071f89eff9853921656959a0d3b3e5fc49de"
Oct 29 22:55:24.687459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:24.687425    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:24.812003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:24.811973    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:24.934888 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:24.934855    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:27.364277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:27.364248    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:29.084511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:29.084476    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:30.345630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:30.345554    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:31.063815 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:31.063757    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:33.101082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:33.101052    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:34.822144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:34.822111    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:36.644605 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:36.644568    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:38.581479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:38.581440    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:40.350637 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:40.350594    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:42.095911 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:42.095879    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:43.879645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:43.879596    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:45.551833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:45.551801    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:47.320762 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:47.320727    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:48.057743 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:55:48.057694    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:55:48.068282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:48.068255    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:48.391178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:48.391099    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"]
Oct 29 22:55:48.429120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:48.429091    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"]
Oct 29 22:55:48.557359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:48.557286    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:48.684442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:48.684396    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:48.905808 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:48.905775    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:49.027490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:49.025683    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:49.576941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:49.576907    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:49.910701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:49.910568    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:50.939201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:50.939170    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:52.572439 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:52.572394    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:53.172642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.172599    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:53.305569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.305479    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:53.391634 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.391599    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj" podUID="058881c3-8417-4e47-b394-0d29112080a5" containerName="payload-processing" containerID="cri-o://45245e311a5f8f73e004ba13cce014569af286ca40e8d64a7abda95a7433be9b" gracePeriod=25
Oct 29 22:55:53.433208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.432493    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8" podUID="2e4c6741-fe7a-4410-9a1b-a47d2021bc58" containerName="payload-pre-processing" containerID="cri-o://5085faa350710b341f15349ae2693b58b4873c6efaebe397233daf8f0630cff1" gracePeriod=25
Oct 29 22:55:53.438932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.438895    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:53.588642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.588475    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:53.636544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.634369    4860 generic.go:356] "Generic (PLEG): container finished" podID="2e4c6741-fe7a-4410-9a1b-a47d2021bc58" containerID="5085faa350710b341f15349ae2693b58b4873c6efaebe397233daf8f0630cff1" exitCode=0
Oct 29 22:55:53.636544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.634522    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8" event={"ID":"2e4c6741-fe7a-4410-9a1b-a47d2021bc58","Type":"ContainerDied","Data":"5085faa350710b341f15349ae2693b58b4873c6efaebe397233daf8f0630cff1"}
Oct 29 22:55:53.636544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.636438    4860 generic.go:356] "Generic (PLEG): container finished" podID="058881c3-8417-4e47-b394-0d29112080a5" containerID="45245e311a5f8f73e004ba13cce014569af286ca40e8d64a7abda95a7433be9b" exitCode=0
Oct 29 22:55:53.636822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.636622    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj" event={"ID":"058881c3-8417-4e47-b394-0d29112080a5","Type":"ContainerDied","Data":"45245e311a5f8f73e004ba13cce014569af286ca40e8d64a7abda95a7433be9b"}
Oct 29 22:55:53.785758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.763260    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:53.845507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.835751    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"
Oct 29 22:55:53.968357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.966977    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058881c3-8417-4e47-b394-0d29112080a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058881c3-8417-4e47-b394-0d29112080a5-plugins-config-volume\") pod \"058881c3-8417-4e47-b394-0d29112080a5\" (UID: \"058881c3-8417-4e47-b394-0d29112080a5\") "
Oct 29 22:55:53.968357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.967342    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058881c3-8417-4e47-b394-0d29112080a5-kube-api-access-4pmgj\" (UniqueName: \"kubernetes.io/projected/058881c3-8417-4e47-b394-0d29112080a5-kube-api-access-4pmgj\") pod \"058881c3-8417-4e47-b394-0d29112080a5\" (UID: \"058881c3-8417-4e47-b394-0d29112080a5\") "
Oct 29 22:55:53.983426 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.976719    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058881c3-8417-4e47-b394-0d29112080a5-plugins-config-volume" pod "058881c3-8417-4e47-b394-0d29112080a5" (UID: "058881c3-8417-4e47-b394-0d29112080a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:54.005475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:53.999176    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058881c3-8417-4e47-b394-0d29112080a5-kube-api-access-4pmgj" pod "058881c3-8417-4e47-b394-0d29112080a5" (UID: "058881c3-8417-4e47-b394-0d29112080a5"). InnerVolumeSpecName "kube-api-access-4pmgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:54.069730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.069699    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pmgj\" (UniqueName: \"kubernetes.io/projected/058881c3-8417-4e47-b394-0d29112080a5-kube-api-access-4pmgj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:55:54.069730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.069729    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058881c3-8417-4e47-b394-0d29112080a5-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:55:54.069979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.069957    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"
Oct 29 22:55:54.170811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.170753    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-plugins-config-volume\") pod \"2e4c6741-fe7a-4410-9a1b-a47d2021bc58\" (UID: \"2e4c6741-fe7a-4410-9a1b-a47d2021bc58\") "
Oct 29 22:55:54.170811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.170803    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-kube-api-access-92zjp\" (UniqueName: \"kubernetes.io/projected/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-kube-api-access-92zjp\") pod \"2e4c6741-fe7a-4410-9a1b-a47d2021bc58\" (UID: \"2e4c6741-fe7a-4410-9a1b-a47d2021bc58\") "
Oct 29 22:55:54.172717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.172689    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-plugins-config-volume" pod "2e4c6741-fe7a-4410-9a1b-a47d2021bc58" (UID: "2e4c6741-fe7a-4410-9a1b-a47d2021bc58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:54.175655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.175627    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-kube-api-access-92zjp" pod "2e4c6741-fe7a-4410-9a1b-a47d2021bc58" (UID: "2e4c6741-fe7a-4410-9a1b-a47d2021bc58"). InnerVolumeSpecName "kube-api-access-92zjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:54.275028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.274934    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:55:54.275028 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.274976    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92zjp\" (UniqueName: \"kubernetes.io/projected/2e4c6741-fe7a-4410-9a1b-a47d2021bc58-kube-api-access-92zjp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:55:54.545148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.545066    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:54.649542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.649511    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"
Oct 29 22:55:54.650104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.649570    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj" event={"ID":"058881c3-8417-4e47-b394-0d29112080a5","Type":"ContainerDied","Data":"7091e2d6ef330c5203dc4261448ba787fc75ee8915132fc277bed6b54fb59080"}
Oct 29 22:55:54.650104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.649653    4860 scope.go:122] "RemoveContainer" containerID="45245e311a5f8f73e004ba13cce014569af286ca40e8d64a7abda95a7433be9b"
Oct 29 22:55:54.652703 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.652414    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"
Oct 29 22:55:54.653069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.652419    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8" event={"ID":"2e4c6741-fe7a-4410-9a1b-a47d2021bc58","Type":"ContainerDied","Data":"861a0bca83735d3cfd5bbc436f1c1a1472a3470708ef4e8a5b0069a1f8677349"}
Oct 29 22:55:54.664162 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.664137    4860 scope.go:122] "RemoveContainer" containerID="5085faa350710b341f15349ae2693b58b4873c6efaebe397233daf8f0630cff1"
Oct 29 22:55:54.716697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.716639    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"]
Oct 29 22:55:54.722024 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.721998    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8c4b02-86d6df458-f5kjj"]
Oct 29 22:55:54.761199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.761167    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"]
Oct 29 22:55:54.766392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:54.766306    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8c4b02-696f4f544c-xbnh8"]
Oct 29 22:55:55.427530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.427473    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:55.547232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.547195    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:55.913377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.913240    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"]
Oct 29 22:55:55.917242 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.917216    4860 cpu_manager.go:414] "containerMap: removing container" podUID="058881c3-8417-4e47-b394-0d29112080a5" containerName="payload-processing"
Oct 29 22:55:55.917461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.917443    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058881c3-8417-4e47-b394-0d29112080a5" containerName="payload-processing"
Oct 29 22:55:55.917679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.917666    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2e4c6741-fe7a-4410-9a1b-a47d2021bc58" containerName="payload-pre-processing"
Oct 29 22:55:55.917802 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.917787    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4c6741-fe7a-4410-9a1b-a47d2021bc58" containerName="payload-pre-processing"
Oct 29 22:55:55.918073 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.918011    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf" containerName="revoke-keys"
Oct 29 22:55:55.918201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.918187    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf" containerName="revoke-keys"
Oct 29 22:55:55.918884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.918862    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="058881c3-8417-4e47-b394-0d29112080a5" containerName="payload-processing"
Oct 29 22:55:55.919174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.919160    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4c6741-fe7a-4410-9a1b-a47d2021bc58" containerName="payload-pre-processing"
Oct 29 22:55:55.919296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.919283    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf" containerName="revoke-keys"
Oct 29 22:55:55.979357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.971434    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"]
Oct 29 22:55:55.979357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:55.971591    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"
Oct 29 22:55:56.082349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.078689    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4796a9-dockercfg-jjrhz\""
Oct 29 22:55:56.082349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.078794    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4796a9\""
Oct 29 22:55:56.136030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.135992    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058881c3-8417-4e47-b394-0d29112080a5" path="/var/lib/kubelet/pods/058881c3-8417-4e47-b394-0d29112080a5/volumes"
Oct 29 22:55:56.136773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.136745    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4c6741-fe7a-4410-9a1b-a47d2021bc58" path="/var/lib/kubelet/pods/2e4c6741-fe7a-4410-9a1b-a47d2021bc58/volumes"
Oct 29 22:55:56.185055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.185004    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52wd8\" (UniqueName: \"kubernetes.io/projected/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-kube-api-access-52wd8\") pod \"payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj\" (UID: \"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"
Oct 29 22:55:56.185244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.185099    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj\" (UID: \"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"
Oct 29 22:55:56.262350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.260888    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:56.293350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.293078    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52wd8\" (UniqueName: \"kubernetes.io/projected/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-kube-api-access-52wd8\") pod \"payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj\" (UID: \"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"
Oct 29 22:55:56.293350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.293185    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj\" (UID: \"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"
Oct 29 22:55:56.311347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.309731    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj\" (UID: \"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"
Oct 29 22:55:56.335430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.325343    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52wd8\" (UniqueName: \"kubernetes.io/projected/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-kube-api-access-52wd8\") pod \"payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj\" (UID: \"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"
Oct 29 22:55:56.449416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.449096    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"
Oct 29 22:55:56.918343 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:55:56.917432    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34181cb_2ee9_42a4_8b1b_0ad9b90b7490.slice/crio-44c91eab7e4b6219b4f880c245a2a84d0160cac3338048b8878ed719c5cf5a92 WatchSource:0}: Error finding container 44c91eab7e4b6219b4f880c245a2a84d0160cac3338048b8878ed719c5cf5a92: Status 404 returned error can't find the container with id 44c91eab7e4b6219b4f880c245a2a84d0160cac3338048b8878ed719c5cf5a92
Oct 29 22:55:56.920569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:56.920540    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"]
Oct 29 22:55:57.485349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.475249    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"]
Oct 29 22:55:57.503350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.488671    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:55:57.507707 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.507673    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"]
Oct 29 22:55:57.529312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.529284    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:55:57.529484 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.529313    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:55:57.540789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.540760    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39f42f74-kr768\" (UID: \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:55:57.541093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.541071    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39f42f74-kr768\" (UID: \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:55:57.541181 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.541158    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhxpw\" (UniqueName: \"kubernetes.io/projected/f70fe7a3-9e71-4add-948e-815c8ef7d84a-kube-api-access-hhxpw\") pod \"maas-api-revoke-sub-39f42f74-kr768\" (UID: \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:55:57.643236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.643189    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39f42f74-kr768\" (UID: \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:55:57.643478 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.643260    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxpw\" (UniqueName: \"kubernetes.io/projected/f70fe7a3-9e71-4add-948e-815c8ef7d84a-kube-api-access-hhxpw\") pod \"maas-api-revoke-sub-39f42f74-kr768\" (UID: \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:55:57.643562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.643486    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39f42f74-kr768\" (UID: \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:55:57.644448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.644424    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39f42f74-kr768\" (UID: \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:55:57.645780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.645759    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39f42f74-kr768\" (UID: \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:55:57.664441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.664416    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhxpw\" (UniqueName: \"kubernetes.io/projected/f70fe7a3-9e71-4add-948e-815c8ef7d84a-kube-api-access-hhxpw\") pod \"maas-api-revoke-sub-39f42f74-kr768\" (UID: \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:55:57.817077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.817000    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:55:57.919932 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.919900    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj" event={"ID":"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490","Type":"ContainerStarted","Data":"5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab"}
Oct 29 22:55:57.920369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.919962    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj" event={"ID":"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490","Type":"ContainerStarted","Data":"44c91eab7e4b6219b4f880c245a2a84d0160cac3338048b8878ed719c5cf5a92"}
Oct 29 22:55:57.921100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.921076    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"
Oct 29 22:55:57.995395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:57.967604    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj" podStartSLOduration=2.967585306 podStartE2EDuration="2.967585306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:57.961384844 +0000 UTC m=+2499.986160976" watchObservedRunningTime="2026-10-29 22:55:57.967585306 +0000 UTC m=+2499.992361440"
Oct 29 22:55:58.050350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:58.050288    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:58.202627 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:55:58.201941    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70fe7a3_9e71_4add_948e_815c8ef7d84a.slice/crio-b630b387c151f66f7da89188f62a4d56a523e564d2fb64571f4b5477dbf3b8b8 WatchSource:0}: Error finding container b630b387c151f66f7da89188f62a4d56a523e564d2fb64571f4b5477dbf3b8b8: Status 404 returned error can't find the container with id b630b387c151f66f7da89188f62a4d56a523e564d2fb64571f4b5477dbf3b8b8
Oct 29 22:55:58.202627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:58.202594    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"]
Oct 29 22:55:58.930086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:58.930052    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768" event={"ID":"f70fe7a3-9e71-4add-948e-815c8ef7d84a","Type":"ContainerStarted","Data":"0e763cb8f0b91a4e8cd31d69bb58ee16beb88a5b2281d7121f4029968ddd5e47"}
Oct 29 22:55:58.930547 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:58.930131    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768" event={"ID":"f70fe7a3-9e71-4add-948e-815c8ef7d84a","Type":"ContainerStarted","Data":"b630b387c151f66f7da89188f62a4d56a523e564d2fb64571f4b5477dbf3b8b8"}
Oct 29 22:55:59.598724 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:59.598633    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:59.944313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:59.944271    4860 generic.go:356] "Generic (PLEG): container finished" podID="f70fe7a3-9e71-4add-948e-815c8ef7d84a" containerID="0e763cb8f0b91a4e8cd31d69bb58ee16beb88a5b2281d7121f4029968ddd5e47" exitCode=0
Oct 29 22:55:59.944761 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:55:59.944347    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768" event={"ID":"f70fe7a3-9e71-4add-948e-815c8ef7d84a","Type":"ContainerDied","Data":"0e763cb8f0b91a4e8cd31d69bb58ee16beb88a5b2281d7121f4029968ddd5e47"}
Oct 29 22:56:00.707613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:00.707560    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:00.956361 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:00.910885    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:01.348346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.347575    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"]
Oct 29 22:56:01.357871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.354502    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"
Oct 29 22:56:01.407401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.406628    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"]
Oct 29 22:56:01.407401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.407025    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7be71d\""
Oct 29 22:56:01.415525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.407825    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7be71d-dockercfg-lfrlr\""
Oct 29 22:56:01.458995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.458963    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91089d4-1a38-4d94-87f8-8131ab078130-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb\" (UID: \"f91089d4-1a38-4d94-87f8-8131ab078130\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"
Oct 29 22:56:01.459158 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.459003    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvqg\" (UniqueName: \"kubernetes.io/projected/f91089d4-1a38-4d94-87f8-8131ab078130-kube-api-access-prvqg\") pod \"payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb\" (UID: \"f91089d4-1a38-4d94-87f8-8131ab078130\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"
Oct 29 22:56:01.561517 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.561474    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91089d4-1a38-4d94-87f8-8131ab078130-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb\" (UID: \"f91089d4-1a38-4d94-87f8-8131ab078130\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"
Oct 29 22:56:01.561706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.561532    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvqg\" (UniqueName: \"kubernetes.io/projected/f91089d4-1a38-4d94-87f8-8131ab078130-kube-api-access-prvqg\") pod \"payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb\" (UID: \"f91089d4-1a38-4d94-87f8-8131ab078130\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"
Oct 29 22:56:01.562180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.562160    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91089d4-1a38-4d94-87f8-8131ab078130-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb\" (UID: \"f91089d4-1a38-4d94-87f8-8131ab078130\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"
Oct 29 22:56:01.597021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.596997    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvqg\" (UniqueName: \"kubernetes.io/projected/f91089d4-1a38-4d94-87f8-8131ab078130-kube-api-access-prvqg\") pod \"payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb\" (UID: \"f91089d4-1a38-4d94-87f8-8131ab078130\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"
Oct 29 22:56:01.636149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.636125    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:56:01.663297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.663221    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-xks-ca\") pod \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\" (UID: \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\") "
Oct 29 22:56:01.663462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.663377    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70fe7a3-9e71-4add-948e-815c8ef7d84a-kube-api-access-hhxpw\" (UniqueName: \"kubernetes.io/projected/f70fe7a3-9e71-4add-948e-815c8ef7d84a-kube-api-access-hhxpw\") pod \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\" (UID: \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\") "
Oct 29 22:56:01.663462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.663411    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-openshift-ca\") pod \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\" (UID: \"f70fe7a3-9e71-4add-948e-815c8ef7d84a\") "
Oct 29 22:56:01.663769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.663747    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-openshift-ca" pod "f70fe7a3-9e71-4add-948e-815c8ef7d84a" (UID: "f70fe7a3-9e71-4add-948e-815c8ef7d84a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:01.664268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.664238    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:56:01.665484 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.665465    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-xks-ca" pod "f70fe7a3-9e71-4add-948e-815c8ef7d84a" (UID: "f70fe7a3-9e71-4add-948e-815c8ef7d84a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:01.665609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.665588    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70fe7a3-9e71-4add-948e-815c8ef7d84a-kube-api-access-hhxpw" pod "f70fe7a3-9e71-4add-948e-815c8ef7d84a" (UID: "f70fe7a3-9e71-4add-948e-815c8ef7d84a"). InnerVolumeSpecName "kube-api-access-hhxpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:01.724556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.724517    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:01.733394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.733371    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"
Oct 29 22:56:01.788404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.767200    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70fe7a3-9e71-4add-948e-815c8ef7d84a-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:56:01.788404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:01.767237    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhxpw\" (UniqueName: \"kubernetes.io/projected/f70fe7a3-9e71-4add-948e-815c8ef7d84a-kube-api-access-hhxpw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:56:02.152816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:02.152785    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"
Oct 29 22:56:02.153197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:02.152822    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768" event={"ID":"f70fe7a3-9e71-4add-948e-815c8ef7d84a","Type":"ContainerDied","Data":"b630b387c151f66f7da89188f62a4d56a523e564d2fb64571f4b5477dbf3b8b8"}
Oct 29 22:56:02.153197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:02.152878    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b630b387c151f66f7da89188f62a4d56a523e564d2fb64571f4b5477dbf3b8b8"
Oct 29 22:56:02.189296 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:02.189227    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"]
Oct 29 22:56:02.190448 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:56:02.190419    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91089d4_1a38_4d94_87f8_8131ab078130.slice/crio-a2d034434e92a46b2a8997f975ca355056fa74f4eaac5142e1047ec23a1315fe WatchSource:0}: Error finding container a2d034434e92a46b2a8997f975ca355056fa74f4eaac5142e1047ec23a1315fe: Status 404 returned error can't find the container with id a2d034434e92a46b2a8997f975ca355056fa74f4eaac5142e1047ec23a1315fe
Oct 29 22:56:02.193430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:02.193411    4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:56:03.166678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:03.166639    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb" event={"ID":"f91089d4-1a38-4d94-87f8-8131ab078130","Type":"ContainerStarted","Data":"a2d034434e92a46b2a8997f975ca355056fa74f4eaac5142e1047ec23a1315fe"}
Oct 29 22:56:03.441506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:03.441468    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:04.180134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:04.180086    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb" event={"ID":"f91089d4-1a38-4d94-87f8-8131ab078130","Type":"ContainerStarted","Data":"eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c"}
Oct 29 22:56:04.180571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:04.180182    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"
Oct 29 22:56:04.223165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:04.223106    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb" podStartSLOduration=1.7012664800000001 podStartE2EDuration="3.223089766s" totalImagesPullingTime="1.521823286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:04.221369958 +0000 UTC m=+2506.246146090" watchObservedRunningTime="2026-10-29 22:56:04.223089766 +0000 UTC m=+2506.247865899"
Oct 29 22:56:05.072252 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:05.072217    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:06.535980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:06.535945    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:07.210826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.210795    4860 generic.go:356] "Generic (PLEG): container finished" podID="d5299767-caa1-4246-b4c1-9217642edfab" containerID="992f3625e72cc437f6d1d36a41ca8cada4b50ee622d49920f870b1d85fb341ba" exitCode=0
Oct 29 22:56:07.211003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.210853    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" event={"ID":"d5299767-caa1-4246-b4c1-9217642edfab","Type":"ContainerDied","Data":"992f3625e72cc437f6d1d36a41ca8cada4b50ee622d49920f870b1d85fb341ba"}
Oct 29 22:56:07.211268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.211249    4860 scope.go:122] "RemoveContainer" containerID="992f3625e72cc437f6d1d36a41ca8cada4b50ee622d49920f870b1d85fb341ba"
Oct 29 22:56:07.213059 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.213028    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:56:07.213136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.213103    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:56:07.213176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.213135    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:56:07.774350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.774310    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"]
Oct 29 22:56:07.775145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.775126    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f70fe7a3-9e71-4add-948e-815c8ef7d84a" containerName="revoke-keys"
Oct 29 22:56:07.775195 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.775147    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70fe7a3-9e71-4add-948e-815c8ef7d84a" containerName="revoke-keys"
Oct 29 22:56:07.775277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.775268    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70fe7a3-9e71-4add-948e-815c8ef7d84a" containerName="revoke-keys"
Oct 29 22:56:07.778521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.778501    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:07.797375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.797350    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"]
Oct 29 22:56:07.809473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.809449    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:56:07.809603 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.809568    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:56:07.911818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.911792    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213603b4-c29d-4889-925a-d13340352ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f264f31b-sh65t\" (UID: \"213603b4-c29d-4889-925a-d13340352ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:07.912003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.911982    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213603b4-c29d-4889-925a-d13340352ba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f264f31b-sh65t\" (UID: \"213603b4-c29d-4889-925a-d13340352ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:07.912056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:07.912035    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8whg\" (UniqueName: \"kubernetes.io/projected/213603b4-c29d-4889-925a-d13340352ba0-kube-api-access-l8whg\") pod \"maas-api-revoke-sub-f264f31b-sh65t\" (UID: \"213603b4-c29d-4889-925a-d13340352ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:08.013809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:08.013765    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8whg\" (UniqueName: \"kubernetes.io/projected/213603b4-c29d-4889-925a-d13340352ba0-kube-api-access-l8whg\") pod \"maas-api-revoke-sub-f264f31b-sh65t\" (UID: \"213603b4-c29d-4889-925a-d13340352ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:08.014030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:08.013961    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213603b4-c29d-4889-925a-d13340352ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f264f31b-sh65t\" (UID: \"213603b4-c29d-4889-925a-d13340352ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:08.014154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:08.014139    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213603b4-c29d-4889-925a-d13340352ba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f264f31b-sh65t\" (UID: \"213603b4-c29d-4889-925a-d13340352ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:08.014655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:08.014632    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213603b4-c29d-4889-925a-d13340352ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f264f31b-sh65t\" (UID: \"213603b4-c29d-4889-925a-d13340352ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:08.016600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:08.016579    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213603b4-c29d-4889-925a-d13340352ba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f264f31b-sh65t\" (UID: \"213603b4-c29d-4889-925a-d13340352ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:08.037277 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:08.037211    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8whg\" (UniqueName: \"kubernetes.io/projected/213603b4-c29d-4889-925a-d13340352ba0-kube-api-access-l8whg\") pod \"maas-api-revoke-sub-f264f31b-sh65t\" (UID: \"213603b4-c29d-4889-925a-d13340352ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:08.040055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:08.040035    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:08.097415 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:08.096581    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:08.226827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:08.226790    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" event={"ID":"d5299767-caa1-4246-b4c1-9217642edfab","Type":"ContainerStarted","Data":"e0341e11e57acb675c60d577c5e22ed67ae7d400467112936b88f746eaa30ee2"}
Oct 29 22:56:08.498046 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:56:08.498014    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213603b4_c29d_4889_925a_d13340352ba0.slice/crio-69aa11f537d1c8dcd237c2f491e7fd8e21d022622e2756accd4da30d65e3b6c6 WatchSource:0}: Error finding container 69aa11f537d1c8dcd237c2f491e7fd8e21d022622e2756accd4da30d65e3b6c6: Status 404 returned error can't find the container with id 69aa11f537d1c8dcd237c2f491e7fd8e21d022622e2756accd4da30d65e3b6c6
Oct 29 22:56:08.500387 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:08.500358    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"]
Oct 29 22:56:09.193521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:09.193479    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:56:09.194023 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:09.193814    4860 patch_prober.go:29] interesting pod/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 29 22:56:09.194023 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:09.193880    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" podUID="d5299767-caa1-4246-b4c1-9217642edfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 29 22:56:09.239276 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:09.239241    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t" event={"ID":"213603b4-c29d-4889-925a-d13340352ba0","Type":"ContainerStarted","Data":"e212690949e3a9dd1a7557ccf529de97a05245bb72dcafa9c37666cddb1b621f"}
Oct 29 22:56:09.239468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:09.239298    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t" event={"ID":"213603b4-c29d-4889-925a-d13340352ba0","Type":"ContainerStarted","Data":"69aa11f537d1c8dcd237c2f491e7fd8e21d022622e2756accd4da30d65e3b6c6"}
Oct 29 22:56:09.296243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:09.296148    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t" podStartSLOduration=2.296134383 podStartE2EDuration="2.296134383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:09.288655753 +0000 UTC m=+2511.313431885" watchObservedRunningTime="2026-10-29 22:56:09.296134383 +0000 UTC m=+2511.320910515"
Oct 29 22:56:09.528249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:09.528202    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:09.948820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:09.948791    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"
Oct 29 22:56:10.194220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:10.194184    4860 patch_prober.go:29] interesting pod/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 29 22:56:10.194635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:10.194245    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" podUID="d5299767-caa1-4246-b4c1-9217642edfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 29 22:56:10.249860 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:10.249830    4860 generic.go:356] "Generic (PLEG): container finished" podID="213603b4-c29d-4889-925a-d13340352ba0" containerID="e212690949e3a9dd1a7557ccf529de97a05245bb72dcafa9c37666cddb1b621f" exitCode=0
Oct 29 22:56:10.250018 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:10.249909    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t" event={"ID":"213603b4-c29d-4889-925a-d13340352ba0","Type":"ContainerDied","Data":"e212690949e3a9dd1a7557ccf529de97a05245bb72dcafa9c37666cddb1b621f"}
Oct 29 22:56:11.193959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.193922    4860 patch_prober.go:29] interesting pod/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 29 22:56:11.194137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.193981    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" podUID="d5299767-caa1-4246-b4c1-9217642edfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 29 22:56:11.293463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.293425    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:11.539958 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.539932    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:11.704194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.703790    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213603b4-c29d-4889-925a-d13340352ba0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213603b4-c29d-4889-925a-d13340352ba0-maas-api-openshift-ca\") pod \"213603b4-c29d-4889-925a-d13340352ba0\" (UID: \"213603b4-c29d-4889-925a-d13340352ba0\") "
Oct 29 22:56:11.704194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.703842    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/213603b4-c29d-4889-925a-d13340352ba0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213603b4-c29d-4889-925a-d13340352ba0-maas-api-xks-ca\") pod \"213603b4-c29d-4889-925a-d13340352ba0\" (UID: \"213603b4-c29d-4889-925a-d13340352ba0\") "
Oct 29 22:56:11.704194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.703937    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213603b4-c29d-4889-925a-d13340352ba0-kube-api-access-l8whg\" (UniqueName: \"kubernetes.io/projected/213603b4-c29d-4889-925a-d13340352ba0-kube-api-access-l8whg\") pod \"213603b4-c29d-4889-925a-d13340352ba0\" (UID: \"213603b4-c29d-4889-925a-d13340352ba0\") "
Oct 29 22:56:11.705122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.704827    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213603b4-c29d-4889-925a-d13340352ba0-maas-api-openshift-ca" pod "213603b4-c29d-4889-925a-d13340352ba0" (UID: "213603b4-c29d-4889-925a-d13340352ba0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:11.706961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.706936    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213603b4-c29d-4889-925a-d13340352ba0-kube-api-access-l8whg" pod "213603b4-c29d-4889-925a-d13340352ba0" (UID: "213603b4-c29d-4889-925a-d13340352ba0"). InnerVolumeSpecName "kube-api-access-l8whg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:11.707828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.707799    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213603b4-c29d-4889-925a-d13340352ba0-maas-api-xks-ca" pod "213603b4-c29d-4889-925a-d13340352ba0" (UID: "213603b4-c29d-4889-925a-d13340352ba0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:11.805545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.805463    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213603b4-c29d-4889-925a-d13340352ba0-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:56:11.805545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.805494    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213603b4-c29d-4889-925a-d13340352ba0-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:56:11.805545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:11.805507    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8whg\" (UniqueName: \"kubernetes.io/projected/213603b4-c29d-4889-925a-d13340352ba0-kube-api-access-l8whg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:56:12.000892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.000853    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"]
Oct 29 22:56:12.002068 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.002049    4860 cpu_manager.go:414] "containerMap: removing container" podUID="213603b4-c29d-4889-925a-d13340352ba0" containerName="revoke-keys"
Oct 29 22:56:12.002124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.002071    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213603b4-c29d-4889-925a-d13340352ba0" containerName="revoke-keys"
Oct 29 22:56:12.002237 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.002226    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="213603b4-c29d-4889-925a-d13340352ba0" containerName="revoke-keys"
Oct 29 22:56:12.006038 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.006014    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.024722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.024699    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"]
Oct 29 22:56:12.032839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.032812    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-02720a-openshift-default-dockercfg-gzwqr\""
Oct 29 22:56:12.109819 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.109735    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-certs\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.109971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.109875    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-token\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.109971 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.109941    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-data\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.110070 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.109973    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kv6m\" (UniqueName: \"kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-kube-api-access-7kv6m\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.110114 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.110101    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istiod-ca-cert\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.110223 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.110210    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-podinfo\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.110264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.110233    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.110368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.110313    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-credential-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.110465 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.110449    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-envoy\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.193980 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.193942    4860 patch_prober.go:29] interesting pod/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 29 22:56:12.194157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.194007    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" podUID="d5299767-caa1-4246-b4c1-9217642edfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 29 22:56:12.212256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.212220    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-token\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.212428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.212272    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-data\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.212428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.212318    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kv6m\" (UniqueName: \"kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-kube-api-access-7kv6m\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.212562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.212441    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istiod-ca-cert\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.212640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.212559    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-podinfo\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.212640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.212588    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.212775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.212639    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-credential-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.212775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.212644    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-data\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.212775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.212732    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-envoy\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.212967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.212801    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-certs\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.212967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.212938    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-credential-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.213211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.213184    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-socket\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.213380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.213352    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istiod-ca-cert\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.213469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.213445    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-certs\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.215405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.215384    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-envoy\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.215498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.215425    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-podinfo\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.235674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.235653    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-token\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.235774 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.235756    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kv6m\" (UniqueName: \"kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-kube-api-access-7kv6m\") pod \"e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") " pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.295345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.294705    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t" event={"ID":"213603b4-c29d-4889-925a-d13340352ba0","Type":"ContainerDied","Data":"69aa11f537d1c8dcd237c2f491e7fd8e21d022622e2756accd4da30d65e3b6c6"}
Oct 29 22:56:12.295345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.294751    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69aa11f537d1c8dcd237c2f491e7fd8e21d022622e2756accd4da30d65e3b6c6"
Oct 29 22:56:12.295345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.294798    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"
Oct 29 22:56:12.323156 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.323130    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:12.519984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.519934    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"]
Oct 29 22:56:12.540791 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.540732    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.548596 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.548566    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"]
Oct 29 22:56:12.699633 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:56:12.699393    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4a8709_3bda_45e4_a052_37b367c4b6c4.slice/crio-dc21b4b3cb5f677c5f3aa22776bf46151e2af97cdc8e2c5e8664226f9a073a44 WatchSource:0}: Error finding container dc21b4b3cb5f677c5f3aa22776bf46151e2af97cdc8e2c5e8664226f9a073a44: Status 404 returned error can't find the container with id dc21b4b3cb5f677c5f3aa22776bf46151e2af97cdc8e2c5e8664226f9a073a44
Oct 29 22:56:12.699947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.699888    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"]
Oct 29 22:56:12.702477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.702293    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:56:12.702653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.702630    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:56:12.702704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.702674    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:56:12.729563 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.729540    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.729655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.729590    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-credential-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.729739 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.729724    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-certs\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.729848 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.729827    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr9kc\" (UniqueName: \"kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-kube-api-access-zr9kc\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.729893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.729865    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88824988-8fe2-4a1b-af86-31647faa8af2-istiod-ca-cert\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.729970 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.729957    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88824988-8fe2-4a1b-af86-31647faa8af2-istio-podinfo\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.730010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.729984    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-istio-token\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.730057 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.730037    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-envoy\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.730113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.730102    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-data\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.832143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.831384    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-certs\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.832143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.831487    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr9kc\" (UniqueName: \"kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-kube-api-access-zr9kc\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.832143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.831511    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88824988-8fe2-4a1b-af86-31647faa8af2-istiod-ca-cert\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.832143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.831561    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88824988-8fe2-4a1b-af86-31647faa8af2-istio-podinfo\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.832143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.831578    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-istio-token\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.832143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.831615    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-envoy\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.832143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.831683    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-data\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.832143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.831746    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.832143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.831785    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-credential-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.832143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.831812    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-certs\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.833089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.832805    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88824988-8fe2-4a1b-af86-31647faa8af2-istiod-ca-cert\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.833461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.833224    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-credential-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.833461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.833415    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-data\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.833585 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.833523    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-socket\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.834677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.834653    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-envoy\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.835443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.835424    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88824988-8fe2-4a1b-af86-31647faa8af2-istio-podinfo\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.852845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.852820    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-istio-token\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.853358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.853307    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr9kc\" (UniqueName: \"kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-kube-api-access-zr9kc\") pod \"e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") " pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:12.862674 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:12.862644    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:13.194782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.194745    4860 patch_prober.go:29] interesting pod/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 29 22:56:13.194987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.194944    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" podUID="d5299767-caa1-4246-b4c1-9217642edfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 29 22:56:13.224030 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:56:13.224003    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88824988_8fe2_4a1b_af86_31647faa8af2.slice/crio-4306c15901429ba781a190bebec921a87a5c0b4556298054dd10fd64d49a5ba5 WatchSource:0}: Error finding container 4306c15901429ba781a190bebec921a87a5c0b4556298054dd10fd64d49a5ba5: Status 404 returned error can't find the container with id 4306c15901429ba781a190bebec921a87a5c0b4556298054dd10fd64d49a5ba5
Oct 29 22:56:13.228051 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.228024    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"]
Oct 29 22:56:13.228230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.228193    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:56:13.228295 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.228281    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:56:13.228369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.228316    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:56:13.311161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.310471    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" event={"ID":"fe4a8709-3bda-45e4-a052-37b367c4b6c4","Type":"ContainerStarted","Data":"e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33"}
Oct 29 22:56:13.311161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.310536    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" event={"ID":"fe4a8709-3bda-45e4-a052-37b367c4b6c4","Type":"ContainerStarted","Data":"dc21b4b3cb5f677c5f3aa22776bf46151e2af97cdc8e2c5e8664226f9a073a44"}
Oct 29 22:56:13.312036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.312007    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx" event={"ID":"88824988-8fe2-4a1b-af86-31647faa8af2","Type":"ContainerStarted","Data":"4306c15901429ba781a190bebec921a87a5c0b4556298054dd10fd64d49a5ba5"}
Oct 29 22:56:13.323390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.323365    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:13.325592 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.325569    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 29 22:56:13.325700 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.325614    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 29 22:56:13.355837 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.355788    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" podStartSLOduration=2.355773948 podStartE2EDuration="2.355773948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:13.353289728 +0000 UTC m=+2515.378065852" watchObservedRunningTime="2026-10-29 22:56:13.355773948 +0000 UTC m=+2515.380550079"
Oct 29 22:56:13.469349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:13.456890    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:14.213350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:14.212601    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:56:14.219486 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:14.214373    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:56:14.219486 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:14.216526    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:56:14.327621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:14.327405    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 29 22:56:14.327621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:14.327478    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 29 22:56:14.336995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:14.336962    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx" event={"ID":"88824988-8fe2-4a1b-af86-31647faa8af2","Type":"ContainerStarted","Data":"fc20f291fe9168359b92731ad417dfc8a7ddfded1b3bff86cbabce60bb0bcb02"}
Oct 29 22:56:14.432347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:14.431675    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx" podStartSLOduration=2.431651925 podStartE2EDuration="2.431651925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:14.416519841 +0000 UTC m=+2516.441295995" watchObservedRunningTime="2026-10-29 22:56:14.431651925 +0000 UTC m=+2516.456428062"
Oct 29 22:56:14.460456 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:14.460420    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:14.863202 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:14.863172    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:14.864193 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:14.864172    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body=
Oct 29 22:56:14.864244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:14.864212    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx" podUID="88824988-8fe2-4a1b-af86-31647faa8af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused"
Oct 29 22:56:15.218350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:15.208695    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"
Oct 29 22:56:15.307357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:15.306373    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:15.325023 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:15.324986    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 29 22:56:15.325272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:15.325045    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 29 22:56:15.863236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:15.863196    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body=
Oct 29 22:56:15.863646 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:15.863262    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx" podUID="88824988-8fe2-4a1b-af86-31647faa8af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused"
Oct 29 22:56:16.324701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:16.323622    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 29 22:56:16.324701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:16.323678    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 29 22:56:16.863497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:16.863463    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body=
Oct 29 22:56:16.864075 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:16.864043    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx" podUID="88824988-8fe2-4a1b-af86-31647faa8af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused"
Oct 29 22:56:17.219014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:17.218982    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:17.326750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:17.326710    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 29 22:56:17.327931 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:17.327897    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 29 22:56:17.863829 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:17.863795    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body=
Oct 29 22:56:17.864232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:17.863852    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx" podUID="88824988-8fe2-4a1b-af86-31647faa8af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused"
Oct 29 22:56:18.059088 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:56:18.058909    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:56:18.323857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:18.323824    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 29 22:56:18.324036 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:18.323880    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 29 22:56:18.863412 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:18.863377    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body=
Oct 29 22:56:18.863590 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:18.863431    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx" podUID="88824988-8fe2-4a1b-af86-31647faa8af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused"
Oct 29 22:56:18.992093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:18.990826    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:19.324319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:19.324281    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 29 22:56:19.324319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:19.324346    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 29 22:56:19.628851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:19.628758    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:19.863809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:19.863771    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused" start-of-body=
Oct 29 22:56:19.863954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:19.863838    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx" podUID="88824988-8fe2-4a1b-af86-31647faa8af2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.57:15021/healthz/ready\": dial tcp 10.132.1.57:15021: connect: connection refused"
Oct 29 22:56:19.928863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:19.928779    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:20.141046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:20.141001    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:20.303749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:20.303715    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:20.588123 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:20.588032    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:20.589694 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:20.589666    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:20.606355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:20.598624    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 22:56:20.815749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:20.815710    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:20.871011 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:20.870920    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:20.871471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:20.871441    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:20.872278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:20.872251    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 22:56:22.603851 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:22.603817    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:23.833978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:23.833948    4860 scope.go:122] "RemoveContainer" containerID="8510b0a9f7eb93d61cb7da386486c950a777cb80c0e1c97b3f97b9ca45102d1f"
Oct 29 22:56:24.203627 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:24.203593    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:24.488544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:24.488459    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:24.628513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:24.628479    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:25.876602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:25.876572    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:26.131962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.131856    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:26.252205 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.252175    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:26.282436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.282408    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"]
Oct 29 22:56:26.288723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.288704    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"
Oct 29 22:56:26.314784 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.314758    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-85862e\""
Oct 29 22:56:26.314954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.314845    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-85862e-dockercfg-fb6pk\""
Oct 29 22:56:26.343377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.343340    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"]
Oct 29 22:56:26.436396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.434054    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkwhm\" (UniqueName: \"kubernetes.io/projected/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-kube-api-access-bkwhm\") pod \"payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf\" (UID: \"604c1f26-0ea0-4e6f-ae6c-259716f8a40b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"
Oct 29 22:56:26.436396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.434250    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf\" (UID: \"604c1f26-0ea0-4e6f-ae6c-259716f8a40b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"
Oct 29 22:56:26.555150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.536712    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwhm\" (UniqueName: \"kubernetes.io/projected/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-kube-api-access-bkwhm\") pod \"payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf\" (UID: \"604c1f26-0ea0-4e6f-ae6c-259716f8a40b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"
Oct 29 22:56:26.555693 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.555662    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf\" (UID: \"604c1f26-0ea0-4e6f-ae6c-259716f8a40b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"
Oct 29 22:56:26.620433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.556926    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf\" (UID: \"604c1f26-0ea0-4e6f-ae6c-259716f8a40b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"
Oct 29 22:56:26.621637 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.621608    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkwhm\" (UniqueName: \"kubernetes.io/projected/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-kube-api-access-bkwhm\") pod \"payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf\" (UID: \"604c1f26-0ea0-4e6f-ae6c-259716f8a40b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"
Oct 29 22:56:26.628650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.628614    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"]
Oct 29 22:56:26.650604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.650550    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"]
Oct 29 22:56:26.650961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.650804    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"
Oct 29 22:56:26.721139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.721057    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hxcn\" (UniqueName: \"kubernetes.io/projected/d57f72e3-b067-4e1e-9295-f6f5f9919255-kube-api-access-5hxcn\") pod \"payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn\" (UID: \"d57f72e3-b067-4e1e-9295-f6f5f9919255\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"
Oct 29 22:56:26.721311 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.721279    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57f72e3-b067-4e1e-9295-f6f5f9919255-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn\" (UID: \"d57f72e3-b067-4e1e-9295-f6f5f9919255\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"
Oct 29 22:56:26.822904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.822841    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57f72e3-b067-4e1e-9295-f6f5f9919255-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn\" (UID: \"d57f72e3-b067-4e1e-9295-f6f5f9919255\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"
Oct 29 22:56:26.823072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.823035    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hxcn\" (UniqueName: \"kubernetes.io/projected/d57f72e3-b067-4e1e-9295-f6f5f9919255-kube-api-access-5hxcn\") pod \"payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn\" (UID: \"d57f72e3-b067-4e1e-9295-f6f5f9919255\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"
Oct 29 22:56:26.823613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.823589    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57f72e3-b067-4e1e-9295-f6f5f9919255-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn\" (UID: \"d57f72e3-b067-4e1e-9295-f6f5f9919255\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"
Oct 29 22:56:26.844630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.844610    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hxcn\" (UniqueName: \"kubernetes.io/projected/d57f72e3-b067-4e1e-9295-f6f5f9919255-kube-api-access-5hxcn\") pod \"payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn\" (UID: \"d57f72e3-b067-4e1e-9295-f6f5f9919255\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"
Oct 29 22:56:26.905388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.905364    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"
Oct 29 22:56:26.975835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:26.974944    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"
Oct 29 22:56:27.210621 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:56:27.210591    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604c1f26_0ea0_4e6f_ae6c_259716f8a40b.slice/crio-0f036ee21988563e8117d35d71949f66d98414943e0e42b10313ff94e266cf5a WatchSource:0}: Error finding container 0f036ee21988563e8117d35d71949f66d98414943e0e42b10313ff94e266cf5a: Status 404 returned error can't find the container with id 0f036ee21988563e8117d35d71949f66d98414943e0e42b10313ff94e266cf5a
Oct 29 22:56:27.214136 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:27.214112    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"]
Oct 29 22:56:27.299454 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:56:27.299423    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57f72e3_b067_4e1e_9295_f6f5f9919255.slice/crio-ffb2633afe6dca09a9cd022de952af9d50f266c284729810a54ae8d8c8d93fc6 WatchSource:0}: Error finding container ffb2633afe6dca09a9cd022de952af9d50f266c284729810a54ae8d8c8d93fc6: Status 404 returned error can't find the container with id ffb2633afe6dca09a9cd022de952af9d50f266c284729810a54ae8d8c8d93fc6
Oct 29 22:56:27.305248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:27.305223    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"]
Oct 29 22:56:27.637144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:27.637054    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf" event={"ID":"604c1f26-0ea0-4e6f-ae6c-259716f8a40b","Type":"ContainerStarted","Data":"4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e"}
Oct 29 22:56:27.637144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:27.637113    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf" event={"ID":"604c1f26-0ea0-4e6f-ae6c-259716f8a40b","Type":"ContainerStarted","Data":"0f036ee21988563e8117d35d71949f66d98414943e0e42b10313ff94e266cf5a"}
Oct 29 22:56:27.637144 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:27.637144    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"
Oct 29 22:56:27.639503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:27.639457    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn" event={"ID":"d57f72e3-b067-4e1e-9295-f6f5f9919255","Type":"ContainerStarted","Data":"61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9"}
Oct 29 22:56:27.639503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:27.639495    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:27.639714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:27.639513    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn" event={"ID":"d57f72e3-b067-4e1e-9295-f6f5f9919255","Type":"ContainerStarted","Data":"ffb2633afe6dca09a9cd022de952af9d50f266c284729810a54ae8d8c8d93fc6"}
Oct 29 22:56:27.639714 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:27.639554    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"
Oct 29 22:56:27.706216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:27.705996    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf" podStartSLOduration=1.7059748959999999 podStartE2EDuration="1.705974896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:27.68184377 +0000 UTC m=+2529.706619931" watchObservedRunningTime="2026-10-29 22:56:27.705974896 +0000 UTC m=+2529.730751037"
Oct 29 22:56:27.734285 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:27.734226    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn" podStartSLOduration=1.734211768 podStartE2EDuration="1.734211768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:27.728749077 +0000 UTC m=+2529.753525210" watchObservedRunningTime="2026-10-29 22:56:27.734211768 +0000 UTC m=+2529.758987898"
Oct 29 22:56:29.176787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:29.176754    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:30.695468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:30.695436    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:31.818809 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:31.818765    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:32.046256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:32.043633    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:33.091031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:33.090976    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:34.301839 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.301802    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:34.439976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.439930    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:34.713963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.713914    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"]
Oct 29 22:56:34.755349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.753715    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"]
Oct 29 22:56:34.755349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.753848    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"
Oct 29 22:56:34.789353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.786501    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a962d7-dockercfg-l5qjc\""
Oct 29 22:56:34.789353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.787233    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a962d7\""
Oct 29 22:56:34.794722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.794688    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jll7\" (UniqueName: \"kubernetes.io/projected/d33876b0-ba2e-48d4-87da-24e68486a8fd-kube-api-access-4jll7\") pod \"payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq\" (UID: \"d33876b0-ba2e-48d4-87da-24e68486a8fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"
Oct 29 22:56:34.821878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.821845    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33876b0-ba2e-48d4-87da-24e68486a8fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq\" (UID: \"d33876b0-ba2e-48d4-87da-24e68486a8fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"
Oct 29 22:56:34.907599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.907549    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"]
Oct 29 22:56:34.928481 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.928449    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"]
Oct 29 22:56:34.929311 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.929107    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"
Oct 29 22:56:34.932305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.932179    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jll7\" (UniqueName: \"kubernetes.io/projected/d33876b0-ba2e-48d4-87da-24e68486a8fd-kube-api-access-4jll7\") pod \"payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq\" (UID: \"d33876b0-ba2e-48d4-87da-24e68486a8fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"
Oct 29 22:56:34.932573 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.932548    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33876b0-ba2e-48d4-87da-24e68486a8fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq\" (UID: \"d33876b0-ba2e-48d4-87da-24e68486a8fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"
Oct 29 22:56:34.933759 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.933717    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33876b0-ba2e-48d4-87da-24e68486a8fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq\" (UID: \"d33876b0-ba2e-48d4-87da-24e68486a8fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"
Oct 29 22:56:34.964305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:34.964223    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jll7\" (UniqueName: \"kubernetes.io/projected/d33876b0-ba2e-48d4-87da-24e68486a8fd-kube-api-access-4jll7\") pod \"payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq\" (UID: \"d33876b0-ba2e-48d4-87da-24e68486a8fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"
Oct 29 22:56:35.034673 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.034641    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d76fbf3e-b816-45d5-876b-284df7a342d2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49\" (UID: \"d76fbf3e-b816-45d5-876b-284df7a342d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"
Oct 29 22:56:35.034835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.034695    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj9zf\" (UniqueName: \"kubernetes.io/projected/d76fbf3e-b816-45d5-876b-284df7a342d2-kube-api-access-fj9zf\") pod \"payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49\" (UID: \"d76fbf3e-b816-45d5-876b-284df7a342d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"
Oct 29 22:56:35.136832 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.136805    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d76fbf3e-b816-45d5-876b-284df7a342d2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49\" (UID: \"d76fbf3e-b816-45d5-876b-284df7a342d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"
Oct 29 22:56:35.136987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.136842    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9zf\" (UniqueName: \"kubernetes.io/projected/d76fbf3e-b816-45d5-876b-284df7a342d2-kube-api-access-fj9zf\") pod \"payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49\" (UID: \"d76fbf3e-b816-45d5-876b-284df7a342d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"
Oct 29 22:56:35.137560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.137540    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d76fbf3e-b816-45d5-876b-284df7a342d2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49\" (UID: \"d76fbf3e-b816-45d5-876b-284df7a342d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"
Oct 29 22:56:35.157279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.157256    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9zf\" (UniqueName: \"kubernetes.io/projected/d76fbf3e-b816-45d5-876b-284df7a342d2-kube-api-access-fj9zf\") pod \"payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49\" (UID: \"d76fbf3e-b816-45d5-876b-284df7a342d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"
Oct 29 22:56:35.183404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.183376    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"
Oct 29 22:56:35.253738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.253708    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"
Oct 29 22:56:35.271953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.271923    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:35.497065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.497035    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"]
Oct 29 22:56:35.619946 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:56:35.619912    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76fbf3e_b816_45d5_876b_284df7a342d2.slice/crio-fe9fbec496c339b332b98d7b06263bd47a2a8d38fb3f21f05f16a32ba975d8bf WatchSource:0}: Error finding container fe9fbec496c339b332b98d7b06263bd47a2a8d38fb3f21f05f16a32ba975d8bf: Status 404 returned error can't find the container with id fe9fbec496c339b332b98d7b06263bd47a2a8d38fb3f21f05f16a32ba975d8bf
Oct 29 22:56:35.626092 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.626059    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"]
Oct 29 22:56:35.749780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.749697    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49" event={"ID":"d76fbf3e-b816-45d5-876b-284df7a342d2","Type":"ContainerStarted","Data":"6fd06f0ebdf6c3f769c1d6187d13249718daa5c75c42af36844f6ab625ba6ba7"}
Oct 29 22:56:35.749780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.749765    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"
Oct 29 22:56:35.750112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.749794    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49" event={"ID":"d76fbf3e-b816-45d5-876b-284df7a342d2","Type":"ContainerStarted","Data":"fe9fbec496c339b332b98d7b06263bd47a2a8d38fb3f21f05f16a32ba975d8bf"}
Oct 29 22:56:35.751519 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.751493    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq" event={"ID":"d33876b0-ba2e-48d4-87da-24e68486a8fd","Type":"ContainerStarted","Data":"508a866267831e7b38642c62cc675f4f61bf1452ed39d55abea9bdd1abc5d60d"}
Oct 29 22:56:35.751647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.751536    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq" event={"ID":"d33876b0-ba2e-48d4-87da-24e68486a8fd","Type":"ContainerStarted","Data":"d65e806ce935b53025020b2e814dab4a91cad76296e46bc2c15d725445e5c0ef"}
Oct 29 22:56:35.751647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.751561    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"
Oct 29 22:56:35.797469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.797414    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49" podStartSLOduration=1.797398016 podStartE2EDuration="1.797398016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:35.793364228 +0000 UTC m=+2537.818140360" watchObservedRunningTime="2026-10-29 22:56:35.797398016 +0000 UTC m=+2537.822174149"
Oct 29 22:56:35.836917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:35.836871    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq" podStartSLOduration=1.836858607 podStartE2EDuration="1.836858607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:35.835859856 +0000 UTC m=+2537.860636006" watchObservedRunningTime="2026-10-29 22:56:35.836858607 +0000 UTC m=+2537.861634737"
Oct 29 22:56:36.865176 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:36.865135    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:38.502706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:38.502663    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:38.626356 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:38.626088    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:38.670942 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:38.670911    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"
Oct 29 22:56:38.672149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:38.672121    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"
Oct 29 22:56:39.033432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:39.033376    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:40.229638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:40.229605    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:41.846619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:41.846590    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:43.025464 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:43.025422    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:43.659159 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:43.659116    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:43.881001 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:43.880960    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:44.148530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:44.148496    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:44.291388 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:44.287557    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:45.291629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:45.291595    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:46.785345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:46.776392    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"
Oct 29 22:56:46.785345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:46.782902    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"
Oct 29 22:56:47.221823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:47.221785    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:48.053204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:48.053169    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:48.061235 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:56:48.061208    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:56:48.203362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:48.203297    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:49.495380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:49.495316    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:49.732716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:49.732681    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:49.860917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:49.860826    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:49.887450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:49.887418    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"]
Oct 29 22:56:49.892557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:49.892539    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"
Oct 29 22:56:49.914287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:49.914256    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"]
Oct 29 22:56:49.921012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:49.920989    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b8f54c\""
Oct 29 22:56:50.016347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:50.015628    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b8f54c-dockercfg-glwcr\""
Oct 29 22:56:50.039442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:50.038543    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce01c801-a808-4a2b-bc03-2538b117742e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz\" (UID: \"ce01c801-a808-4a2b-bc03-2538b117742e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"
Oct 29 22:56:50.039442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:50.038712    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg52v\" (UniqueName: \"kubernetes.io/projected/ce01c801-a808-4a2b-bc03-2538b117742e-kube-api-access-fg52v\") pod \"payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz\" (UID: \"ce01c801-a808-4a2b-bc03-2538b117742e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"
Oct 29 22:56:50.155813 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:50.155700    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce01c801-a808-4a2b-bc03-2538b117742e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz\" (UID: \"ce01c801-a808-4a2b-bc03-2538b117742e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"
Oct 29 22:56:50.156187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:50.156162    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg52v\" (UniqueName: \"kubernetes.io/projected/ce01c801-a808-4a2b-bc03-2538b117742e-kube-api-access-fg52v\") pod \"payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz\" (UID: \"ce01c801-a808-4a2b-bc03-2538b117742e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"
Oct 29 22:56:50.158536 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:50.158497    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce01c801-a808-4a2b-bc03-2538b117742e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz\" (UID: \"ce01c801-a808-4a2b-bc03-2538b117742e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"
Oct 29 22:56:50.233532 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:50.233498    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg52v\" (UniqueName: \"kubernetes.io/projected/ce01c801-a808-4a2b-bc03-2538b117742e-kube-api-access-fg52v\") pod \"payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz\" (UID: \"ce01c801-a808-4a2b-bc03-2538b117742e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"
Oct 29 22:56:50.508194 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:50.508162    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"
Oct 29 22:56:50.839878 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:56:50.839835    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce01c801_a808_4a2b_bc03_2538b117742e.slice/crio-124ad9b44c716946937b1583612d1019d80a2d703965b5767c32bd53b615ce86 WatchSource:0}: Error finding container 124ad9b44c716946937b1583612d1019d80a2d703965b5767c32bd53b615ce86: Status 404 returned error can't find the container with id 124ad9b44c716946937b1583612d1019d80a2d703965b5767c32bd53b615ce86
Oct 29 22:56:50.845363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:50.845311    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"]
Oct 29 22:56:51.083391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.083309    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:51.158503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.158048    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz" event={"ID":"ce01c801-a808-4a2b-bc03-2538b117742e","Type":"ContainerStarted","Data":"3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239"}
Oct 29 22:56:51.158503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.158122    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz" event={"ID":"ce01c801-a808-4a2b-bc03-2538b117742e","Type":"ContainerStarted","Data":"124ad9b44c716946937b1583612d1019d80a2d703965b5767c32bd53b615ce86"}
Oct 29 22:56:51.159826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.159538    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"
Oct 29 22:56:51.201204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.201169    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"]
Oct 29 22:56:51.215264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.213792    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:51.218161 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.217920    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz" podStartSLOduration=2.217904143 podStartE2EDuration="2.217904143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:51.207559584 +0000 UTC m=+2553.232335743" watchObservedRunningTime="2026-10-29 22:56:51.217904143 +0000 UTC m=+2553.242680274"
Oct 29 22:56:51.231084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.231060    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"]
Oct 29 22:56:51.231443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.231414    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:56:51.232516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.232493    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:56:51.280177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.280150    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e823990-ptdgp\" (UID: \"5689700a-6385-4b1a-beda-af609bf6cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:51.280313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.280190    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e823990-ptdgp\" (UID: \"5689700a-6385-4b1a-beda-af609bf6cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:51.280313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.280289    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p5wz\" (UniqueName: \"kubernetes.io/projected/5689700a-6385-4b1a-beda-af609bf6cec4-kube-api-access-4p5wz\") pod \"maas-api-revoke-sub-4e823990-ptdgp\" (UID: \"5689700a-6385-4b1a-beda-af609bf6cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:51.382150 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.382109    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e823990-ptdgp\" (UID: \"5689700a-6385-4b1a-beda-af609bf6cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:51.382383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.382190    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e823990-ptdgp\" (UID: \"5689700a-6385-4b1a-beda-af609bf6cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:51.382383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.382288    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p5wz\" (UniqueName: \"kubernetes.io/projected/5689700a-6385-4b1a-beda-af609bf6cec4-kube-api-access-4p5wz\") pod \"maas-api-revoke-sub-4e823990-ptdgp\" (UID: \"5689700a-6385-4b1a-beda-af609bf6cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:51.382902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.382877    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e823990-ptdgp\" (UID: \"5689700a-6385-4b1a-beda-af609bf6cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:51.384787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.384758    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e823990-ptdgp\" (UID: \"5689700a-6385-4b1a-beda-af609bf6cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:51.408105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.408073    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p5wz\" (UniqueName: \"kubernetes.io/projected/5689700a-6385-4b1a-beda-af609bf6cec4-kube-api-access-4p5wz\") pod \"maas-api-revoke-sub-4e823990-ptdgp\" (UID: \"5689700a-6385-4b1a-beda-af609bf6cec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:51.421615 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.421535    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"]
Oct 29 22:56:51.428171 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.428149    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:51.442895 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.442870    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"]
Oct 29 22:56:51.483754 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.483719    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q\" (UID: \"1fc13d1d-3237-4625-a3fd-3292ded0251c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:51.483996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.483974    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5d5m\" (UniqueName: \"kubernetes.io/projected/1fc13d1d-3237-4625-a3fd-3292ded0251c-kube-api-access-c5d5m\") pod \"maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q\" (UID: \"1fc13d1d-3237-4625-a3fd-3292ded0251c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:51.484180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.484163    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q\" (UID: \"1fc13d1d-3237-4625-a3fd-3292ded0251c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:51.535301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.535266    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:51.586186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.585666    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5d5m\" (UniqueName: \"kubernetes.io/projected/1fc13d1d-3237-4625-a3fd-3292ded0251c-kube-api-access-c5d5m\") pod \"maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q\" (UID: \"1fc13d1d-3237-4625-a3fd-3292ded0251c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:51.586372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.586242    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q\" (UID: \"1fc13d1d-3237-4625-a3fd-3292ded0251c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:51.586372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.586351    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q\" (UID: \"1fc13d1d-3237-4625-a3fd-3292ded0251c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:51.587112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.587091    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q\" (UID: \"1fc13d1d-3237-4625-a3fd-3292ded0251c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:51.588870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.588832    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q\" (UID: \"1fc13d1d-3237-4625-a3fd-3292ded0251c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:51.609706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.609683    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5d5m\" (UniqueName: \"kubernetes.io/projected/1fc13d1d-3237-4625-a3fd-3292ded0251c-kube-api-access-c5d5m\") pod \"maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q\" (UID: \"1fc13d1d-3237-4625-a3fd-3292ded0251c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:51.747931 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.747902    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:51.852619 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:56:51.852591    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5689700a_6385_4b1a_beda_af609bf6cec4.slice/crio-e4d0d9969c41fbcd1a07f7fc470350749bfe2362fa6a674773f40e8e2d187ca9 WatchSource:0}: Error finding container e4d0d9969c41fbcd1a07f7fc470350749bfe2362fa6a674773f40e8e2d187ca9: Status 404 returned error can't find the container with id e4d0d9969c41fbcd1a07f7fc470350749bfe2362fa6a674773f40e8e2d187ca9
Oct 29 22:56:51.852822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:51.852796    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"]
Oct 29 22:56:52.061029 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:56:52.060996    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc13d1d_3237_4625_a3fd_3292ded0251c.slice/crio-6ec3dc37bbeb48220ef5f62e9e13f2010ba603250afe6ae22230a0ebab518f03 WatchSource:0}: Error finding container 6ec3dc37bbeb48220ef5f62e9e13f2010ba603250afe6ae22230a0ebab518f03: Status 404 returned error can't find the container with id 6ec3dc37bbeb48220ef5f62e9e13f2010ba603250afe6ae22230a0ebab518f03
Oct 29 22:56:52.065302 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:52.065277    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"]
Oct 29 22:56:52.169142 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:52.169101    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp" event={"ID":"5689700a-6385-4b1a-beda-af609bf6cec4","Type":"ContainerStarted","Data":"e4d0d9969c41fbcd1a07f7fc470350749bfe2362fa6a674773f40e8e2d187ca9"}
Oct 29 22:56:52.171145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:52.171113    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q" event={"ID":"1fc13d1d-3237-4625-a3fd-3292ded0251c","Type":"ContainerStarted","Data":"6ec3dc37bbeb48220ef5f62e9e13f2010ba603250afe6ae22230a0ebab518f03"}
Oct 29 22:56:52.903587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:52.903552    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:53.183288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:53.183258    4860 generic.go:356] "Generic (PLEG): container finished" podID="5689700a-6385-4b1a-beda-af609bf6cec4" containerID="23a252cbc4648c4278e756ac30f478c2c0fd58720c8239e56b02f3837fc1fbe4" exitCode=0
Oct 29 22:56:53.183506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:53.183358    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp" event={"ID":"5689700a-6385-4b1a-beda-af609bf6cec4","Type":"ContainerDied","Data":"23a252cbc4648c4278e756ac30f478c2c0fd58720c8239e56b02f3837fc1fbe4"}
Oct 29 22:56:53.185291 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:53.185263    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q" event={"ID":"1fc13d1d-3237-4625-a3fd-3292ded0251c","Type":"ContainerStarted","Data":"669c72661c6ea46b39c86c74db4bac9b0dada230230e8b5438684c89f76a5aef"}
Oct 29 22:56:53.267442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:53.267386    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q" podStartSLOduration=2.26737228 podStartE2EDuration="2.26737228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:53.266635806 +0000 UTC m=+2555.291411959" watchObservedRunningTime="2026-10-29 22:56:53.26737228 +0000 UTC m=+2555.292148404"
Oct 29 22:56:54.196049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.196018    4860 generic.go:356] "Generic (PLEG): container finished" podID="1fc13d1d-3237-4625-a3fd-3292ded0251c" containerID="669c72661c6ea46b39c86c74db4bac9b0dada230230e8b5438684c89f76a5aef" exitCode=0
Oct 29 22:56:54.196489 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.196103    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q" event={"ID":"1fc13d1d-3237-4625-a3fd-3292ded0251c","Type":"ContainerDied","Data":"669c72661c6ea46b39c86c74db4bac9b0dada230230e8b5438684c89f76a5aef"}
Oct 29 22:56:54.398093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.398068    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:54.436834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.436808    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-xks-ca\") pod \"5689700a-6385-4b1a-beda-af609bf6cec4\" (UID: \"5689700a-6385-4b1a-beda-af609bf6cec4\") "
Oct 29 22:56:54.437012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.436889    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5689700a-6385-4b1a-beda-af609bf6cec4-kube-api-access-4p5wz\" (UniqueName: \"kubernetes.io/projected/5689700a-6385-4b1a-beda-af609bf6cec4-kube-api-access-4p5wz\") pod \"5689700a-6385-4b1a-beda-af609bf6cec4\" (UID: \"5689700a-6385-4b1a-beda-af609bf6cec4\") "
Oct 29 22:56:54.437091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.437021    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-openshift-ca\") pod \"5689700a-6385-4b1a-beda-af609bf6cec4\" (UID: \"5689700a-6385-4b1a-beda-af609bf6cec4\") "
Oct 29 22:56:54.438786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.438753    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-openshift-ca" pod "5689700a-6385-4b1a-beda-af609bf6cec4" (UID: "5689700a-6385-4b1a-beda-af609bf6cec4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:54.439623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.439595    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-xks-ca" pod "5689700a-6385-4b1a-beda-af609bf6cec4" (UID: "5689700a-6385-4b1a-beda-af609bf6cec4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:54.440264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.440223    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5689700a-6385-4b1a-beda-af609bf6cec4-kube-api-access-4p5wz" pod "5689700a-6385-4b1a-beda-af609bf6cec4" (UID: "5689700a-6385-4b1a-beda-af609bf6cec4"). InnerVolumeSpecName "kube-api-access-4p5wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:54.471853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.471817    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:54.540597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.540544    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:56:54.540597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.540589    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5689700a-6385-4b1a-beda-af609bf6cec4-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:56:54.540924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:54.540606    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p5wz\" (UniqueName: \"kubernetes.io/projected/5689700a-6385-4b1a-beda-af609bf6cec4-kube-api-access-4p5wz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:56:55.207618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.207592    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"
Oct 29 22:56:55.208086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.207632    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp" event={"ID":"5689700a-6385-4b1a-beda-af609bf6cec4","Type":"ContainerDied","Data":"e4d0d9969c41fbcd1a07f7fc470350749bfe2362fa6a674773f40e8e2d187ca9"}
Oct 29 22:56:55.208086 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.207677    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4d0d9969c41fbcd1a07f7fc470350749bfe2362fa6a674773f40e8e2d187ca9"
Oct 29 22:56:55.410615 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.410590    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:55.453668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.453640    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc13d1d-3237-4625-a3fd-3292ded0251c-kube-api-access-c5d5m\" (UniqueName: \"kubernetes.io/projected/1fc13d1d-3237-4625-a3fd-3292ded0251c-kube-api-access-c5d5m\") pod \"1fc13d1d-3237-4625-a3fd-3292ded0251c\" (UID: \"1fc13d1d-3237-4625-a3fd-3292ded0251c\") "
Oct 29 22:56:55.453845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.453825    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-openshift-ca\") pod \"1fc13d1d-3237-4625-a3fd-3292ded0251c\" (UID: \"1fc13d1d-3237-4625-a3fd-3292ded0251c\") "
Oct 29 22:56:55.454074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.454048    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-xks-ca\") pod \"1fc13d1d-3237-4625-a3fd-3292ded0251c\" (UID: \"1fc13d1d-3237-4625-a3fd-3292ded0251c\") "
Oct 29 22:56:55.454242 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.454199    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-openshift-ca" pod "1fc13d1d-3237-4625-a3fd-3292ded0251c" (UID: "1fc13d1d-3237-4625-a3fd-3292ded0251c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:55.454938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.454912    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:56:55.457298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.457121    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc13d1d-3237-4625-a3fd-3292ded0251c-kube-api-access-c5d5m" pod "1fc13d1d-3237-4625-a3fd-3292ded0251c" (UID: "1fc13d1d-3237-4625-a3fd-3292ded0251c"). InnerVolumeSpecName "kube-api-access-c5d5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:55.457298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.457136    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-xks-ca" pod "1fc13d1d-3237-4625-a3fd-3292ded0251c" (UID: "1fc13d1d-3237-4625-a3fd-3292ded0251c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:55.556677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.556636    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fc13d1d-3237-4625-a3fd-3292ded0251c-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:56:55.556677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:55.556679    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5d5m\" (UniqueName: \"kubernetes.io/projected/1fc13d1d-3237-4625-a3fd-3292ded0251c-kube-api-access-c5d5m\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:56:56.155533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:56.155497    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:56.235025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:56.234986    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"
Oct 29 22:56:56.235483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:56.235018    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q" event={"ID":"1fc13d1d-3237-4625-a3fd-3292ded0251c","Type":"ContainerDied","Data":"6ec3dc37bbeb48220ef5f62e9e13f2010ba603250afe6ae22230a0ebab518f03"}
Oct 29 22:56:56.235483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:56.235173    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ec3dc37bbeb48220ef5f62e9e13f2010ba603250afe6ae22230a0ebab518f03"
Oct 29 22:56:57.790015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:57.789977    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:59.415256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:56:59.415218    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:01.663626 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.663588    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"]
Oct 29 22:57:01.664469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.664452    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1fc13d1d-3237-4625-a3fd-3292ded0251c" containerName="revoke-keys"
Oct 29 22:57:01.664469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.664469    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc13d1d-3237-4625-a3fd-3292ded0251c" containerName="revoke-keys"
Oct 29 22:57:01.664552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.664490    4860 cpu_manager.go:414] "containerMap: removing container" podUID="5689700a-6385-4b1a-beda-af609bf6cec4" containerName="revoke-keys"
Oct 29 22:57:01.664552 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.664496    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5689700a-6385-4b1a-beda-af609bf6cec4" containerName="revoke-keys"
Oct 29 22:57:01.664638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.664628    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc13d1d-3237-4625-a3fd-3292ded0251c" containerName="revoke-keys"
Oct 29 22:57:01.664672 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.664642    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5689700a-6385-4b1a-beda-af609bf6cec4" containerName="revoke-keys"
Oct 29 22:57:01.669824 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.669806    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:01.688221 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.688195    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"]
Oct 29 22:57:01.692109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.692089    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:01.692904 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.692882    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:57:01.696411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.696386    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:01.774720 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.762260    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkrh6\" (UniqueName: \"kubernetes.io/projected/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-kube-api-access-rkrh6\") pod \"maas-api-revoke-sub-5e23de74-w9ps5\" (UID: \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:01.784315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.784269    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e23de74-w9ps5\" (UID: \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:01.784771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.784736    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e23de74-w9ps5\" (UID: \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:01.889940 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.889903    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkrh6\" (UniqueName: \"kubernetes.io/projected/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-kube-api-access-rkrh6\") pod \"maas-api-revoke-sub-5e23de74-w9ps5\" (UID: \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:01.890128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.889989    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e23de74-w9ps5\" (UID: \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:01.890216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.890202    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e23de74-w9ps5\" (UID: \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:01.890769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.890745    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e23de74-w9ps5\" (UID: \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:01.892607 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.892582    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e23de74-w9ps5\" (UID: \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:01.910956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.910931    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkrh6\" (UniqueName: \"kubernetes.io/projected/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-kube-api-access-rkrh6\") pod \"maas-api-revoke-sub-5e23de74-w9ps5\" (UID: \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:01.986606 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:01.986581    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:02.290032 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:57:02.289996    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742c9aa7_4d6a_4200_b408_66b7c8f09d7f.slice/crio-a05162ba7ead5065842f267ba2557894e564030ae842051465829208768878f1 WatchSource:0}: Error finding container a05162ba7ead5065842f267ba2557894e564030ae842051465829208768878f1: Status 404 returned error can't find the container with id a05162ba7ead5065842f267ba2557894e564030ae842051465829208768878f1
Oct 29 22:57:02.293591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:02.293565    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"]
Oct 29 22:57:03.188760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:03.188728    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"
Oct 29 22:57:03.301006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:03.300969    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5" event={"ID":"742c9aa7-4d6a-4200-b408-66b7c8f09d7f","Type":"ContainerStarted","Data":"605bde6ee38f50beff6ee1fb2b34a59fed049474101b35416f33be12652b195f"}
Oct 29 22:57:03.301166 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:03.301025    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5" event={"ID":"742c9aa7-4d6a-4200-b408-66b7c8f09d7f","Type":"ContainerStarted","Data":"a05162ba7ead5065842f267ba2557894e564030ae842051465829208768878f1"}
Oct 29 22:57:03.345260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:03.345208    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5" podStartSLOduration=2.34519425 podStartE2EDuration="2.34519425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:03.342260846 +0000 UTC m=+2565.367036978" watchObservedRunningTime="2026-10-29 22:57:03.34519425 +0000 UTC m=+2565.369970381"
Oct 29 22:57:03.348543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:03.348519    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:04.313366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:04.313333    4860 generic.go:356] "Generic (PLEG): container finished" podID="742c9aa7-4d6a-4200-b408-66b7c8f09d7f" containerID="605bde6ee38f50beff6ee1fb2b34a59fed049474101b35416f33be12652b195f" exitCode=0
Oct 29 22:57:04.313767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:04.313414    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5" event={"ID":"742c9aa7-4d6a-4200-b408-66b7c8f09d7f","Type":"ContainerDied","Data":"605bde6ee38f50beff6ee1fb2b34a59fed049474101b35416f33be12652b195f"}
Oct 29 22:57:04.881350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:04.881180    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:05.505012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:05.504986    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:05.582054 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:05.581970    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-kube-api-access-rkrh6\" (UniqueName: \"kubernetes.io/projected/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-kube-api-access-rkrh6\") pod \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\" (UID: \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\") "
Oct 29 22:57:05.582209 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:05.582120    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-openshift-ca\") pod \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\" (UID: \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\") "
Oct 29 22:57:05.582209 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:05.582191    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-xks-ca\") pod \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\" (UID: \"742c9aa7-4d6a-4200-b408-66b7c8f09d7f\") "
Oct 29 22:57:05.582760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:05.582730    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-openshift-ca" pod "742c9aa7-4d6a-4200-b408-66b7c8f09d7f" (UID: "742c9aa7-4d6a-4200-b408-66b7c8f09d7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:05.583168 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:05.583136    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:05.584608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:05.584575    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-kube-api-access-rkrh6" pod "742c9aa7-4d6a-4200-b408-66b7c8f09d7f" (UID: "742c9aa7-4d6a-4200-b408-66b7c8f09d7f"). InnerVolumeSpecName "kube-api-access-rkrh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:05.584608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:05.584582    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-xks-ca" pod "742c9aa7-4d6a-4200-b408-66b7c8f09d7f" (UID: "742c9aa7-4d6a-4200-b408-66b7c8f09d7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:05.684833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:05.684804    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:05.684833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:05.684830    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkrh6\" (UniqueName: \"kubernetes.io/projected/742c9aa7-4d6a-4200-b408-66b7c8f09d7f-kube-api-access-rkrh6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:06.333521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:06.333489    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5" event={"ID":"742c9aa7-4d6a-4200-b408-66b7c8f09d7f","Type":"ContainerDied","Data":"a05162ba7ead5065842f267ba2557894e564030ae842051465829208768878f1"}
Oct 29 22:57:06.333668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:06.333539    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a05162ba7ead5065842f267ba2557894e564030ae842051465829208768878f1"
Oct 29 22:57:06.333668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:06.333507    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"
Oct 29 22:57:06.624966 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:06.624929    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:08.234269 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:08.234238    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:09.475270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:09.475237    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:09.799221 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:09.799131    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:11.459495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:11.459454    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:11.583344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:11.577883    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:11.693924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:11.693885    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:13.155364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:13.155294    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:13.872948 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:13.872915    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:14.672650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:14.672619    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:14.880445 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:14.880414    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:15.168645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:15.168609    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:16.620621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:16.620590    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:18.056982 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:57:18.056943    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:57:18.296192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:18.296161    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:19.972499 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:19.972467    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:22.271734 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:22.271676    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:24.288069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:24.288034    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:25.795509 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:25.795477    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:27.589992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:27.589959    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:29.180366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:29.180317    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:30.776377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:30.776303    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:31.048072 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:31.047975    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:31.189785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:31.189748    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:31.856486 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:31.856439    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:32.136542 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.136452    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"]
Oct 29 22:57:32.145656 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.145629    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"]
Oct 29 22:57:32.146662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.146640    4860 cpu_manager.go:414] "containerMap: removing container" podUID="742c9aa7-4d6a-4200-b408-66b7c8f09d7f" containerName="revoke-keys"
Oct 29 22:57:32.146662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.146660    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742c9aa7-4d6a-4200-b408-66b7c8f09d7f" containerName="revoke-keys"
Oct 29 22:57:32.146818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.146801    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="742c9aa7-4d6a-4200-b408-66b7c8f09d7f" containerName="revoke-keys"
Oct 29 22:57:32.150947 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.150925    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:32.167742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.167717    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:32.170529 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.170506    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:57:32.172289 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.172265    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"]
Oct 29 22:57:32.241655 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.241619    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdrqz\" (UniqueName: \"kubernetes.io/projected/e91bf163-2b0e-4788-ab6a-05aed58cf339-kube-api-access-pdrqz\") pod \"maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg\" (UID: \"e91bf163-2b0e-4788-ab6a-05aed58cf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:32.242008 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.241985    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg\" (UID: \"e91bf163-2b0e-4788-ab6a-05aed58cf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:32.242359 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.242307    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg\" (UID: \"e91bf163-2b0e-4788-ab6a-05aed58cf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:32.255622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.255585    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:32.343133 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.343097    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg\" (UID: \"e91bf163-2b0e-4788-ab6a-05aed58cf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:32.343307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.343205    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg\" (UID: \"e91bf163-2b0e-4788-ab6a-05aed58cf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:32.343307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.343245    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdrqz\" (UniqueName: \"kubernetes.io/projected/e91bf163-2b0e-4788-ab6a-05aed58cf339-kube-api-access-pdrqz\") pod \"maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg\" (UID: \"e91bf163-2b0e-4788-ab6a-05aed58cf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:32.343961 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.343939    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg\" (UID: \"e91bf163-2b0e-4788-ab6a-05aed58cf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:32.345545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.345513    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg\" (UID: \"e91bf163-2b0e-4788-ab6a-05aed58cf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:32.364108 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.364084    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdrqz\" (UniqueName: \"kubernetes.io/projected/e91bf163-2b0e-4788-ab6a-05aed58cf339-kube-api-access-pdrqz\") pod \"maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg\" (UID: \"e91bf163-2b0e-4788-ab6a-05aed58cf339\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:32.368472 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.368445    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:32.478916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.478611    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:32.509845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.509812    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:32.637566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:32.637251    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:33.110377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:33.107157    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"]
Oct 29 22:57:33.666007 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:33.665931    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg" event={"ID":"e91bf163-2b0e-4788-ab6a-05aed58cf339","Type":"ContainerStarted","Data":"d0a3d9f177844dd68348fef059cc99456cc33ea00643cbb339d2e86df604d341"}
Oct 29 22:57:33.666128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:33.666031    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg" event={"ID":"e91bf163-2b0e-4788-ab6a-05aed58cf339","Type":"ContainerStarted","Data":"87e46e4536a0d22b8fb0e47e5ece8cadb69238c5c73b30e308c51efce1e4611a"}
Oct 29 22:57:33.713288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:33.713238    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg" podStartSLOduration=1.713225381 podStartE2EDuration="1.713225381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:33.70701111 +0000 UTC m=+2595.731787241" watchObservedRunningTime="2026-10-29 22:57:33.713225381 +0000 UTC m=+2595.738001511"
Oct 29 22:57:34.560498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:34.560462    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:34.679236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:34.679202    4860 generic.go:356] "Generic (PLEG): container finished" podID="e91bf163-2b0e-4788-ab6a-05aed58cf339" containerID="d0a3d9f177844dd68348fef059cc99456cc33ea00643cbb339d2e86df604d341" exitCode=0
Oct 29 22:57:34.679510 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:34.679297    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg" event={"ID":"e91bf163-2b0e-4788-ab6a-05aed58cf339","Type":"ContainerDied","Data":"d0a3d9f177844dd68348fef059cc99456cc33ea00643cbb339d2e86df604d341"}
Oct 29 22:57:35.870861 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:35.870826    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:35.963262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:35.963181    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e91bf163-2b0e-4788-ab6a-05aed58cf339-kube-api-access-pdrqz\" (UniqueName: \"kubernetes.io/projected/e91bf163-2b0e-4788-ab6a-05aed58cf339-kube-api-access-pdrqz\") pod \"e91bf163-2b0e-4788-ab6a-05aed58cf339\" (UID: \"e91bf163-2b0e-4788-ab6a-05aed58cf339\") "
Oct 29 22:57:35.963262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:35.963230    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-openshift-ca\") pod \"e91bf163-2b0e-4788-ab6a-05aed58cf339\" (UID: \"e91bf163-2b0e-4788-ab6a-05aed58cf339\") "
Oct 29 22:57:35.963525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:35.963265    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-xks-ca\") pod \"e91bf163-2b0e-4788-ab6a-05aed58cf339\" (UID: \"e91bf163-2b0e-4788-ab6a-05aed58cf339\") "
Oct 29 22:57:35.963704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:35.963672    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-openshift-ca" pod "e91bf163-2b0e-4788-ab6a-05aed58cf339" (UID: "e91bf163-2b0e-4788-ab6a-05aed58cf339"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:35.963953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:35.963929    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:35.965913 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:35.965879    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91bf163-2b0e-4788-ab6a-05aed58cf339-kube-api-access-pdrqz" pod "e91bf163-2b0e-4788-ab6a-05aed58cf339" (UID: "e91bf163-2b0e-4788-ab6a-05aed58cf339"). InnerVolumeSpecName "kube-api-access-pdrqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:35.966047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:35.965942    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-xks-ca" pod "e91bf163-2b0e-4788-ab6a-05aed58cf339" (UID: "e91bf163-2b0e-4788-ab6a-05aed58cf339"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:36.065865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:36.065836    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdrqz\" (UniqueName: \"kubernetes.io/projected/e91bf163-2b0e-4788-ab6a-05aed58cf339-kube-api-access-pdrqz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:36.065865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:36.065863    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91bf163-2b0e-4788-ab6a-05aed58cf339-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:36.326502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:36.326407    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:36.454353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:36.452965    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:36.700124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:36.700094    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"
Oct 29 22:57:36.700282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:36.700139    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg" event={"ID":"e91bf163-2b0e-4788-ab6a-05aed58cf339","Type":"ContainerDied","Data":"87e46e4536a0d22b8fb0e47e5ece8cadb69238c5c73b30e308c51efce1e4611a"}
Oct 29 22:57:36.700282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:36.700194    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87e46e4536a0d22b8fb0e47e5ece8cadb69238c5c73b30e308c51efce1e4611a"
Oct 29 22:57:36.863653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:36.863596    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:37.013788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.013696    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:37.141407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.141182    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj" podUID="a34181cb-2ee9-42a4-8b1b-0ad9b90b7490" containerName="payload-processing" containerID="cri-o://5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab" gracePeriod=25
Oct 29 22:57:37.536122 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.536061    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"
Oct 29 22:57:37.621543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.621506    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-kube-api-access-52wd8\" (UniqueName: \"kubernetes.io/projected/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-kube-api-access-52wd8\") pod \"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490\" (UID: \"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490\") "
Oct 29 22:57:37.621880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.621858    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-plugins-config-volume\") pod \"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490\" (UID: \"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490\") "
Oct 29 22:57:37.622243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.622220    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-plugins-config-volume" pod "a34181cb-2ee9-42a4-8b1b-0ad9b90b7490" (UID: "a34181cb-2ee9-42a4-8b1b-0ad9b90b7490"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:37.622708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.622684    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:37.624290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.624250    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-kube-api-access-52wd8" pod "a34181cb-2ee9-42a4-8b1b-0ad9b90b7490" (UID: "a34181cb-2ee9-42a4-8b1b-0ad9b90b7490"). InnerVolumeSpecName "kube-api-access-52wd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:37.771348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.767704    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52wd8\" (UniqueName: \"kubernetes.io/projected/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490-kube-api-access-52wd8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:37.860926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.860790    4860 generic.go:356] "Generic (PLEG): container finished" podID="a34181cb-2ee9-42a4-8b1b-0ad9b90b7490" containerID="5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab" exitCode=0
Oct 29 22:57:37.861169 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.861141    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj" event={"ID":"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490","Type":"ContainerDied","Data":"5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab"}
Oct 29 22:57:37.861264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.861246    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj" event={"ID":"a34181cb-2ee9-42a4-8b1b-0ad9b90b7490","Type":"ContainerDied","Data":"44c91eab7e4b6219b4f880c245a2a84d0160cac3338048b8878ed719c5cf5a92"}
Oct 29 22:57:37.861352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.861298    4860 scope.go:122] "RemoveContainer" containerID="5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab"
Oct 29 22:57:37.861490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.861467    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"
Oct 29 22:57:37.878110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.878087    4860 scope.go:122] "RemoveContainer" containerID="5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab"
Oct 29 22:57:37.878386 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:57:37.878365    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab\": container with ID starting with 5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab not found: ID does not exist" containerID="5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab"
Oct 29 22:57:37.878461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.878395    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab"} err="failed to get container status \"5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab\": rpc error: code = NotFound desc = could not find container \"5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab\": container with ID starting with 5cdc125d45dd6e856a02ae510d0d95030f5c065d7099c374449f0c929c4ae6ab not found: ID does not exist"
Oct 29 22:57:37.923364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.923297    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"]
Oct 29 22:57:37.933078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:37.933049    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4796a9-6c55db49cd-kkpxj"]
Oct 29 22:57:38.120012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.119933    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a34181cb-2ee9-42a4-8b1b-0ad9b90b7490" path="/var/lib/kubelet/pods/a34181cb-2ee9-42a4-8b1b-0ad9b90b7490/volumes"
Oct 29 22:57:38.568527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.568492    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:38.693872 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.693841    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:38.725135 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.725102    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"]
Oct 29 22:57:38.726069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.726044    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a34181cb-2ee9-42a4-8b1b-0ad9b90b7490" containerName="payload-processing"
Oct 29 22:57:38.726069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.726065    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a34181cb-2ee9-42a4-8b1b-0ad9b90b7490" containerName="payload-processing"
Oct 29 22:57:38.726293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.726079    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e91bf163-2b0e-4788-ab6a-05aed58cf339" containerName="revoke-keys"
Oct 29 22:57:38.726293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.726086    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e91bf163-2b0e-4788-ab6a-05aed58cf339" containerName="revoke-keys"
Oct 29 22:57:38.726293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.726242    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a34181cb-2ee9-42a4-8b1b-0ad9b90b7490" containerName="payload-processing"
Oct 29 22:57:38.726293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.726250    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e91bf163-2b0e-4788-ab6a-05aed58cf339" containerName="revoke-keys"
Oct 29 22:57:38.731978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.731956    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"
Oct 29 22:57:38.750737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.748396    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"]
Oct 29 22:57:38.753556 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.753535    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-83f120\""
Oct 29 22:57:38.753721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.753671    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-83f120-dockercfg-wh9mm\""
Oct 29 22:57:38.781909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.781883    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"]
Oct 29 22:57:38.786960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.786934    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsnw6\" (UniqueName: \"kubernetes.io/projected/ef4421ef-787c-44cb-b6e5-e582c584582f-kube-api-access-bsnw6\") pod \"payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct\" (UID: \"ef4421ef-787c-44cb-b6e5-e582c584582f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"
Oct 29 22:57:38.787060 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.787015    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4421ef-787c-44cb-b6e5-e582c584582f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct\" (UID: \"ef4421ef-787c-44cb-b6e5-e582c584582f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"
Oct 29 22:57:38.798438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.788300    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"
Oct 29 22:57:38.817962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.817932    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"]
Oct 29 22:57:38.912396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.908128    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb59757-ce69-45bb-9bea-fd9701270b86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r\" (UID: \"fbb59757-ce69-45bb-9bea-fd9701270b86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"
Oct 29 22:57:38.912396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.908213    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsnw6\" (UniqueName: \"kubernetes.io/projected/ef4421ef-787c-44cb-b6e5-e582c584582f-kube-api-access-bsnw6\") pod \"payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct\" (UID: \"ef4421ef-787c-44cb-b6e5-e582c584582f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"
Oct 29 22:57:38.912396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.908283    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4421ef-787c-44cb-b6e5-e582c584582f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct\" (UID: \"ef4421ef-787c-44cb-b6e5-e582c584582f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"
Oct 29 22:57:38.912396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.908535    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f97fl\" (UniqueName: \"kubernetes.io/projected/fbb59757-ce69-45bb-9bea-fd9701270b86-kube-api-access-f97fl\") pod \"payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r\" (UID: \"fbb59757-ce69-45bb-9bea-fd9701270b86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"
Oct 29 22:57:38.912396 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:38.909580    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4421ef-787c-44cb-b6e5-e582c584582f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct\" (UID: \"ef4421ef-787c-44cb-b6e5-e582c584582f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"
Oct 29 22:57:39.045348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:39.045288    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsnw6\" (UniqueName: \"kubernetes.io/projected/ef4421ef-787c-44cb-b6e5-e582c584582f-kube-api-access-bsnw6\") pod \"payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct\" (UID: \"ef4421ef-787c-44cb-b6e5-e582c584582f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"
Oct 29 22:57:39.056346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:39.049778    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f97fl\" (UniqueName: \"kubernetes.io/projected/fbb59757-ce69-45bb-9bea-fd9701270b86-kube-api-access-f97fl\") pod \"payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r\" (UID: \"fbb59757-ce69-45bb-9bea-fd9701270b86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"
Oct 29 22:57:39.056346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:39.049969    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb59757-ce69-45bb-9bea-fd9701270b86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r\" (UID: \"fbb59757-ce69-45bb-9bea-fd9701270b86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"
Oct 29 22:57:39.056346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:39.051577    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb59757-ce69-45bb-9bea-fd9701270b86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r\" (UID: \"fbb59757-ce69-45bb-9bea-fd9701270b86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"
Oct 29 22:57:39.076859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:39.075672    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"
Oct 29 22:57:39.167683 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:39.165270    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f97fl\" (UniqueName: \"kubernetes.io/projected/fbb59757-ce69-45bb-9bea-fd9701270b86-kube-api-access-f97fl\") pod \"payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r\" (UID: \"fbb59757-ce69-45bb-9bea-fd9701270b86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"
Oct 29 22:57:39.436418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:39.428762    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"
Oct 29 22:57:40.097363 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:57:40.096549    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4421ef_787c_44cb_b6e5_e582c584582f.slice/crio-cb403c4393bd46dee301ca5be836baaf70b2549c4236f999bc0258ae2823f80c WatchSource:0}: Error finding container cb403c4393bd46dee301ca5be836baaf70b2549c4236f999bc0258ae2823f80c: Status 404 returned error can't find the container with id cb403c4393bd46dee301ca5be836baaf70b2549c4236f999bc0258ae2823f80c
Oct 29 22:57:40.304735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:40.304696    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"]
Oct 29 22:57:40.565477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:40.565423    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct" event={"ID":"ef4421ef-787c-44cb-b6e5-e582c584582f","Type":"ContainerStarted","Data":"cb403c4393bd46dee301ca5be836baaf70b2549c4236f999bc0258ae2823f80c"}
Oct 29 22:57:40.575399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:40.573393    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.776647 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:40.776591    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.869521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:40.776917    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.869521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:40.777292    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Oct 29 22:57:40.869521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:40.825938    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main" probeResult="failure" output="Get \"https://10.132.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.869521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:40.827392    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:57:40.869521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:40.841403    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.869521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:40.845395    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.966386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:40.961416    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.966386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:40.963589    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Oct 29 22:57:41.131864 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.131831    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:41.283403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.254875    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"]
Oct 29 22:57:41.374345 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.362875    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:57:41.419469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.419421    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.591075 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.589953    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.591075 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.590764    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:57:41.643364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.614694    4860 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 29 22:57:41.862401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.854301    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.890858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.870891    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:57:41.890858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.879756    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:57:41.890858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.880652    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.890858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.881274    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main" probeResult="failure" output="Get \"https://10.132.0.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.890858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.881887    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:57:41.890858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.882742    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:57:41.890858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.882947    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:57:41.890858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.883004    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:57:41.890858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.883061    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:57:41.890858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.875449    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.906578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.892368    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:57:41.906578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.897472    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.906578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.897792    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:57:41.994071 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:41.994039    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 22:57:42.031355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.023606    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 22:57:42.031355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.028445    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:57:42.031355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.029215    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 22:57:42.031355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.029759    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:57:42.085857 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.085822    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r" event={"ID":"fbb59757-ce69-45bb-9bea-fd9701270b86","Type":"ContainerStarted","Data":"a218f25bac28986caaffec050f07f1fbe1ffc4649693fb56f2c6b531d21b8c8a"}
Oct 29 22:57:42.086811 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.086778    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"
Oct 29 22:57:42.088543 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.088512    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:42.089863 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.089841    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:57:42.093902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.093884    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 22:57:42.099672 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.099652    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 22:57:42.100976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.100959    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 22:57:42.131584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.131426    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 22:57:42.131584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.131465    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:57:42.131584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.131496    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 22:57:42.131584 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.131519    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 22:57:42.412305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.412172    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r" podStartSLOduration=4.412155089 podStartE2EDuration="4.412155089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:42.403684495 +0000 UTC m=+2604.428460646" watchObservedRunningTime="2026-10-29 22:57:42.412155089 +0000 UTC m=+2604.436931221"
Oct 29 22:57:42.791833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:42.791798    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:43.114750 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:43.114031    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r" event={"ID":"fbb59757-ce69-45bb-9bea-fd9701270b86","Type":"ContainerStarted","Data":"319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb"}
Oct 29 22:57:43.120069 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:43.120040    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct" event={"ID":"ef4421ef-787c-44cb-b6e5-e582c584582f","Type":"ContainerStarted","Data":"445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13"}
Oct 29 22:57:43.122021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:43.122001    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"
Oct 29 22:57:43.166418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:43.166371    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct" podStartSLOduration=5.166359015 podStartE2EDuration="5.166359015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:43.162439937 +0000 UTC m=+2605.187216088" watchObservedRunningTime="2026-10-29 22:57:43.166359015 +0000 UTC m=+2605.191135136"
Oct 29 22:57:44.440589 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:44.440549    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:45.157395 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.157344    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"]
Oct 29 22:57:45.162733 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.162709    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:45.182204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.182172    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"]
Oct 29 22:57:45.188623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.188594    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:57:45.188623 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.188609    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:45.262362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.262283    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw2g2\" (UniqueName: \"kubernetes.io/projected/2cda08f6-a831-44d0-b9ac-ff84b67331d9-kube-api-access-lw2g2\") pod \"maas-api-revoke-sub-bedbb1d7-m86b9\" (UID: \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:45.262537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.262418    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bedbb1d7-m86b9\" (UID: \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:45.262816 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.262788    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bedbb1d7-m86b9\" (UID: \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:45.364191 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.364161    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw2g2\" (UniqueName: \"kubernetes.io/projected/2cda08f6-a831-44d0-b9ac-ff84b67331d9-kube-api-access-lw2g2\") pod \"maas-api-revoke-sub-bedbb1d7-m86b9\" (UID: \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:45.364422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.364207    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bedbb1d7-m86b9\" (UID: \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:45.364422 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.364405    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bedbb1d7-m86b9\" (UID: \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:45.365022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.365001    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bedbb1d7-m86b9\" (UID: \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:45.366755 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.366731    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bedbb1d7-m86b9\" (UID: \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:45.384973 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.384949    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw2g2\" (UniqueName: \"kubernetes.io/projected/2cda08f6-a831-44d0-b9ac-ff84b67331d9-kube-api-access-lw2g2\") pod \"maas-api-revoke-sub-bedbb1d7-m86b9\" (UID: \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:45.481691 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.481661    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:45.797011 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:57:45.796981    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cda08f6_a831_44d0_b9ac_ff84b67331d9.slice/crio-a61d8567fc81e49253b36866700d824757ed223481866de679cdd5b4a75c105d WatchSource:0}: Error finding container a61d8567fc81e49253b36866700d824757ed223481866de679cdd5b4a75c105d: Status 404 returned error can't find the container with id a61d8567fc81e49253b36866700d824757ed223481866de679cdd5b4a75c105d
Oct 29 22:57:45.801442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:45.801414    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"]
Oct 29 22:57:46.080401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:46.080375    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:46.182496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:46.182452    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9" event={"ID":"2cda08f6-a831-44d0-b9ac-ff84b67331d9","Type":"ContainerStarted","Data":"a425528dba76db0abf9c20e1989688d9bcdba98e072d5186844c1c235a2bb1cf"}
Oct 29 22:57:46.182715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:46.182521    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9" event={"ID":"2cda08f6-a831-44d0-b9ac-ff84b67331d9","Type":"ContainerStarted","Data":"a61d8567fc81e49253b36866700d824757ed223481866de679cdd5b4a75c105d"}
Oct 29 22:57:46.226335 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:46.226272    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9" podStartSLOduration=1.2262581479999999 podStartE2EDuration="1.226258148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:46.223436228 +0000 UTC m=+2608.248212354" watchObservedRunningTime="2026-10-29 22:57:46.226258148 +0000 UTC m=+2608.251034279"
Oct 29 22:57:47.193640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:47.193608    4860 generic.go:356] "Generic (PLEG): container finished" podID="2cda08f6-a831-44d0-b9ac-ff84b67331d9" containerID="a425528dba76db0abf9c20e1989688d9bcdba98e072d5186844c1c235a2bb1cf" exitCode=0
Oct 29 22:57:47.194095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:47.193651    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9" event={"ID":"2cda08f6-a831-44d0-b9ac-ff84b67331d9","Type":"ContainerDied","Data":"a425528dba76db0abf9c20e1989688d9bcdba98e072d5186844c1c235a2bb1cf"}
Oct 29 22:57:47.633964 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:47.633884    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:48.058072 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:57:48.058035    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:57:48.394320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:48.394289    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:48.423999 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:48.423972    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-openshift-ca\") pod \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\" (UID: \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\") "
Oct 29 22:57:48.424237 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:48.424215    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cda08f6-a831-44d0-b9ac-ff84b67331d9-kube-api-access-lw2g2\" (UniqueName: \"kubernetes.io/projected/2cda08f6-a831-44d0-b9ac-ff84b67331d9-kube-api-access-lw2g2\") pod \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\" (UID: \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\") "
Oct 29 22:57:48.424393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:48.424372    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-xks-ca\") pod \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\" (UID: \"2cda08f6-a831-44d0-b9ac-ff84b67331d9\") "
Oct 29 22:57:48.424507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:48.424374    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-openshift-ca" pod "2cda08f6-a831-44d0-b9ac-ff84b67331d9" (UID: "2cda08f6-a831-44d0-b9ac-ff84b67331d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:48.425065 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:48.425040    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:48.426506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:48.426481    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cda08f6-a831-44d0-b9ac-ff84b67331d9-kube-api-access-lw2g2" pod "2cda08f6-a831-44d0-b9ac-ff84b67331d9" (UID: "2cda08f6-a831-44d0-b9ac-ff84b67331d9"). InnerVolumeSpecName "kube-api-access-lw2g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:48.426506 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:48.426493    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-xks-ca" pod "2cda08f6-a831-44d0-b9ac-ff84b67331d9" (UID: "2cda08f6-a831-44d0-b9ac-ff84b67331d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:48.526688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:48.526659    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw2g2\" (UniqueName: \"kubernetes.io/projected/2cda08f6-a831-44d0-b9ac-ff84b67331d9-kube-api-access-lw2g2\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:48.526688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:48.526688    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cda08f6-a831-44d0-b9ac-ff84b67331d9-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:49.216010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:49.215976    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9" event={"ID":"2cda08f6-a831-44d0-b9ac-ff84b67331d9","Type":"ContainerDied","Data":"a61d8567fc81e49253b36866700d824757ed223481866de679cdd5b4a75c105d"}
Oct 29 22:57:49.216010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:49.215999    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"
Oct 29 22:57:49.216222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:49.216024    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a61d8567fc81e49253b36866700d824757ed223481866de679cdd5b4a75c105d"
Oct 29 22:57:49.252498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:49.252473    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:50.595569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:50.595528    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:50.786118 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:50.786089    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:50.914990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:50.914911    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:51.085411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:51.085373    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"]
Oct 29 22:57:51.168406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:51.151963    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c89770e-sm5sl"]
Oct 29 22:57:51.190179 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:51.190144    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:51.325265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:51.325225    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:51.345055 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:51.345025    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"]
Oct 29 22:57:51.357452 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:51.357423    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c87bf1-4tn2q"]
Oct 29 22:57:51.443880 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:51.443737    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:51.800711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:51.800628    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:52.119392 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:52.119271    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc13d1d-3237-4625-a3fd-3292ded0251c" path="/var/lib/kubelet/pods/1fc13d1d-3237-4625-a3fd-3292ded0251c/volumes"
Oct 29 22:57:52.119983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:52.119956    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e64b5e6-bae2-4768-bb6a-159914642d59" path="/var/lib/kubelet/pods/6e64b5e6-bae2-4768-bb6a-159914642d59/volumes"
Oct 29 22:57:53.242062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:53.242029    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:54.035113 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:54.035082    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"]
Oct 29 22:57:54.045894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:54.045863    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48712fe8-jn9kx"]
Oct 29 22:57:54.118048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:54.118018    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7a1462-6c14-4963-b9ee-246e986d876a" path="/var/lib/kubelet/pods/8a7a1462-6c14-4963-b9ee-246e986d876a/volumes"
Oct 29 22:57:54.147049 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:54.147027    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"
Oct 29 22:57:54.853449 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:54.853418    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:55.158351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.158265    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"
Oct 29 22:57:55.503582 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.503550    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"]
Oct 29 22:57:55.505476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.505449    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2cda08f6-a831-44d0-b9ac-ff84b67331d9" containerName="revoke-keys"
Oct 29 22:57:55.505650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.505637    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cda08f6-a831-44d0-b9ac-ff84b67331d9" containerName="revoke-keys"
Oct 29 22:57:55.506052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.506035    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cda08f6-a831-44d0-b9ac-ff84b67331d9" containerName="revoke-keys"
Oct 29 22:57:55.550927 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.550893    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"]
Oct 29 22:57:55.551085 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.550999    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:55.567431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.567401    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:55.567822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.567521    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:57:55.659668 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.659634    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf4477c3-g7fkw\" (UID: \"ed203352-bea6-4522-a312-a9cf7a03582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:55.659834 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.659774    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf4477c3-g7fkw\" (UID: \"ed203352-bea6-4522-a312-a9cf7a03582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:55.660012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.659983    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fwcl\" (UniqueName: \"kubernetes.io/projected/ed203352-bea6-4522-a312-a9cf7a03582e-kube-api-access-4fwcl\") pod \"maas-api-revoke-sub-bf4477c3-g7fkw\" (UID: \"ed203352-bea6-4522-a312-a9cf7a03582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:55.761384 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.761297    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fwcl\" (UniqueName: \"kubernetes.io/projected/ed203352-bea6-4522-a312-a9cf7a03582e-kube-api-access-4fwcl\") pod \"maas-api-revoke-sub-bf4477c3-g7fkw\" (UID: \"ed203352-bea6-4522-a312-a9cf7a03582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:55.761618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.761600    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf4477c3-g7fkw\" (UID: \"ed203352-bea6-4522-a312-a9cf7a03582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:55.761748 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.761731    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf4477c3-g7fkw\" (UID: \"ed203352-bea6-4522-a312-a9cf7a03582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:55.762477 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.762445    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf4477c3-g7fkw\" (UID: \"ed203352-bea6-4522-a312-a9cf7a03582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:55.764428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.764401    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf4477c3-g7fkw\" (UID: \"ed203352-bea6-4522-a312-a9cf7a03582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:55.782640 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.782613    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fwcl\" (UniqueName: \"kubernetes.io/projected/ed203352-bea6-4522-a312-a9cf7a03582e-kube-api-access-4fwcl\") pod \"maas-api-revoke-sub-bf4477c3-g7fkw\" (UID: \"ed203352-bea6-4522-a312-a9cf7a03582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:55.868985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:55.868960    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:56.170655 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:57:56.170623    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded203352_bea6_4522_a312_a9cf7a03582e.slice/crio-1acd45ab5a558e025b704f5139a5fba2d21a19ff00726c57e6a1e0e1c22b307b WatchSource:0}: Error finding container 1acd45ab5a558e025b704f5139a5fba2d21a19ff00726c57e6a1e0e1c22b307b: Status 404 returned error can't find the container with id 1acd45ab5a558e025b704f5139a5fba2d21a19ff00726c57e6a1e0e1c22b307b
Oct 29 22:57:56.175019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:56.174991    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"]
Oct 29 22:57:56.352663 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:56.352633    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw" event={"ID":"ed203352-bea6-4522-a312-a9cf7a03582e","Type":"ContainerStarted","Data":"1acd45ab5a558e025b704f5139a5fba2d21a19ff00726c57e6a1e0e1c22b307b"}
Oct 29 22:57:56.387151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:56.387127    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:56.944167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:56.944137    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"]
Oct 29 22:57:56.949231 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:56.949213    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:56.995045 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:56.995006    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"]
Oct 29 22:57:57.089557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.089526    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.089726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.089576    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.089726 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.089605    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.089820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.089750    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.089820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.089788    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.089916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.089895    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.089959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.089948    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4hv\" (UniqueName: \"kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-kube-api-access-ss4hv\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.090074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.090059    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.090149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.090097    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.191035 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191006    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4hv\" (UniqueName: \"kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-kube-api-access-ss4hv\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.191209 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191072    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.191209 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191098    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.191437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191278    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.191437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191336    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.191437 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191378    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.191616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191456    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.191616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191473    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.191616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191489    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.191616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191592    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.191818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191784    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-certs\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.191957 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191933    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-data\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.192027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.191993    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-credential-socket\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.192220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.192200    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istiod-ca-cert\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.193468 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.193444    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-envoy\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.193827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.193808    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-podinfo\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.212402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.212308    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-token\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.212498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.212433    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4hv\" (UniqueName: \"kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-kube-api-access-ss4hv\") pod \"e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.265418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.265394    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:57.366483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.366450    4860 generic.go:356] "Generic (PLEG): container finished" podID="ed203352-bea6-4522-a312-a9cf7a03582e" containerID="c249a1965397b2d33bea46f2a3ca5fd136f4e151d7cfe7831802b5dd8175003d" exitCode=0
Oct 29 22:57:57.366645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.366510    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw" event={"ID":"ed203352-bea6-4522-a312-a9cf7a03582e","Type":"ContainerDied","Data":"c249a1965397b2d33bea46f2a3ca5fd136f4e151d7cfe7831802b5dd8175003d"}
Oct 29 22:57:57.595673 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:57:57.595647    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d6d3a9_38ae_481f_827b_80d1f3524ab6.slice/crio-5f9ed791b36de07a9ec40b911c09d76999db9738ff9bc02bc15fd98029afdfa5 WatchSource:0}: Error finding container 5f9ed791b36de07a9ec40b911c09d76999db9738ff9bc02bc15fd98029afdfa5: Status 404 returned error can't find the container with id 5f9ed791b36de07a9ec40b911c09d76999db9738ff9bc02bc15fd98029afdfa5
Oct 29 22:57:57.598561 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.598523    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:57:57.598682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.598613    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:57:57.598682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.598658    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:57:57.600362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:57.600340    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"]
Oct 29 22:57:58.025357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.024939    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:58.037813 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.037782    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"]
Oct 29 22:57:58.043745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.043721    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41fb3d7-lt25l"]
Oct 29 22:57:58.146581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.146552    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:58.151207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.151175    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1fb025-6a6e-46a3-98db-e36c4ed6ea15" path="/var/lib/kubelet/pods/fd1fb025-6a6e-46a3-98db-e36c4ed6ea15/volumes"
Oct 29 22:57:58.285812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.285727    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:58.392375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.392281    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c" event={"ID":"c0d6d3a9-38ae-481f-827b-80d1f3524ab6","Type":"ContainerStarted","Data":"30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62"}
Oct 29 22:57:58.392679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.392652    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c" event={"ID":"c0d6d3a9-38ae-481f-827b-80d1f3524ab6","Type":"ContainerStarted","Data":"5f9ed791b36de07a9ec40b911c09d76999db9738ff9bc02bc15fd98029afdfa5"}
Oct 29 22:57:58.395828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.395789    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:58.846017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.845885    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c" podStartSLOduration=2.845868205 podStartE2EDuration="2.845868205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:58.499213997 +0000 UTC m=+2620.523990145" watchObservedRunningTime="2026-10-29 22:57:58.845868205 +0000 UTC m=+2620.870644336"
Oct 29 22:57:58.847788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.847762    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"]
Oct 29 22:57:58.854597 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.854577    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:57:58.913679 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.897901    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"]
Oct 29 22:57:58.913959 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:58.913927    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:59.023062 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.023033    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed203352-bea6-4522-a312-a9cf7a03582e-kube-api-access-4fwcl\" (UniqueName: \"kubernetes.io/projected/ed203352-bea6-4522-a312-a9cf7a03582e-kube-api-access-4fwcl\") pod \"ed203352-bea6-4522-a312-a9cf7a03582e\" (UID: \"ed203352-bea6-4522-a312-a9cf7a03582e\") "
Oct 29 22:57:59.023268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.023166    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-openshift-ca\") pod \"ed203352-bea6-4522-a312-a9cf7a03582e\" (UID: \"ed203352-bea6-4522-a312-a9cf7a03582e\") "
Oct 29 22:57:59.023448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.023430    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-xks-ca\") pod \"ed203352-bea6-4522-a312-a9cf7a03582e\" (UID: \"ed203352-bea6-4522-a312-a9cf7a03582e\") "
Oct 29 22:57:59.023711 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.023688    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5\" (UID: \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:57:59.023810 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.023758    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qw6g\" (UniqueName: \"kubernetes.io/projected/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-kube-api-access-5qw6g\") pod \"maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5\" (UID: \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:57:59.023893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.023877    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5\" (UID: \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:57:59.026498 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.026475    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed203352-bea6-4522-a312-a9cf7a03582e-kube-api-access-4fwcl" pod "ed203352-bea6-4522-a312-a9cf7a03582e" (UID: "ed203352-bea6-4522-a312-a9cf7a03582e"). InnerVolumeSpecName "kube-api-access-4fwcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:59.026819 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.026787    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-openshift-ca" pod "ed203352-bea6-4522-a312-a9cf7a03582e" (UID: "ed203352-bea6-4522-a312-a9cf7a03582e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:59.028457 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.028430    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-xks-ca" pod "ed203352-bea6-4522-a312-a9cf7a03582e" (UID: "ed203352-bea6-4522-a312-a9cf7a03582e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:59.137341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.129651    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5\" (UID: \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:57:59.137341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.130010    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5\" (UID: \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:57:59.137341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.130078    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qw6g\" (UniqueName: \"kubernetes.io/projected/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-kube-api-access-5qw6g\") pod \"maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5\" (UID: \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:57:59.137341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.130242    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:59.137341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.130260    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fwcl\" (UniqueName: \"kubernetes.io/projected/ed203352-bea6-4522-a312-a9cf7a03582e-kube-api-access-4fwcl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:59.137341 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.130274    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed203352-bea6-4522-a312-a9cf7a03582e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:57:59.138410 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.138180    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5\" (UID: \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:57:59.142441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.142027    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5\" (UID: \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:57:59.154163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.154130    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qw6g\" (UniqueName: \"kubernetes.io/projected/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-kube-api-access-5qw6g\") pod \"maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5\" (UID: \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:57:59.203818 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.203783    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:57:59.296757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.281791    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:57:59.296757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.295729    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Oct 29 22:57:59.296757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.295796    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c" podUID="c0d6d3a9-38ae-481f-827b-80d1f3524ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Oct 29 22:57:59.407967 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.407940    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"
Oct 29 22:57:59.408153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.407976    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw" event={"ID":"ed203352-bea6-4522-a312-a9cf7a03582e","Type":"ContainerDied","Data":"1acd45ab5a558e025b704f5139a5fba2d21a19ff00726c57e6a1e0e1c22b307b"}
Oct 29 22:57:59.408153 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.408046    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1acd45ab5a558e025b704f5139a5fba2d21a19ff00726c57e6a1e0e1c22b307b"
Oct 29 22:57:59.527176 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:57:59.527146    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e797a3_5609_40fc_bba4_9e1ab4cdec50.slice/crio-9f3fe972c868c19214cc356e38a909984a6f7213c3cf9f93e9183ddd743457d4 WatchSource:0}: Error finding container 9f3fe972c868c19214cc356e38a909984a6f7213c3cf9f93e9183ddd743457d4: Status 404 returned error can't find the container with id 9f3fe972c868c19214cc356e38a909984a6f7213c3cf9f93e9183ddd743457d4
Oct 29 22:57:59.530469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.530438    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"]
Oct 29 22:57:59.893825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:57:59.893740    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:00.265940 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.265903    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Oct 29 22:58:00.266371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.265960    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c" podUID="c0d6d3a9-38ae-481f-827b-80d1f3524ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Oct 29 22:58:00.419836 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.419798    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5" event={"ID":"f7e797a3-5609-40fc-bba4-9e1ab4cdec50","Type":"ContainerStarted","Data":"bd520a3cffa2126811b554290c65a09bb2823d9c440de7c69bec4cc8db1a60c4"}
Oct 29 22:58:00.420023 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.419862    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5" event={"ID":"f7e797a3-5609-40fc-bba4-9e1ab4cdec50","Type":"ContainerStarted","Data":"9f3fe972c868c19214cc356e38a909984a6f7213c3cf9f93e9183ddd743457d4"}
Oct 29 22:58:00.466954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.466902    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5" podStartSLOduration=2.466888635 podStartE2EDuration="2.466888635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:00.460040283 +0000 UTC m=+2622.484816415" watchObservedRunningTime="2026-10-29 22:58:00.466888635 +0000 UTC m=+2622.491664766"
Oct 29 22:58:00.855511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.855482    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"]
Oct 29 22:58:00.856337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.856311    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ed203352-bea6-4522-a312-a9cf7a03582e" containerName="revoke-keys"
Oct 29 22:58:00.856408 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.856345    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed203352-bea6-4522-a312-a9cf7a03582e" containerName="revoke-keys"
Oct 29 22:58:00.856549 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.856539    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed203352-bea6-4522-a312-a9cf7a03582e" containerName="revoke-keys"
Oct 29 22:58:00.860404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.860378    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:00.912544 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.912512    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"]
Oct 29 22:58:00.995370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.995225    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8wm\" (UniqueName: \"kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-kube-api-access-cs8wm\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:00.995370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.995347    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-envoy\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:00.995639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.995434    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-data\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:00.995639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.995485    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-token\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:00.995639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.995521    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-certs\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:00.995639 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.995616    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-credential-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:00.995887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.995650    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istiod-ca-cert\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:00.995887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.995750    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:00.995887 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:00.995788    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-podinfo\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.031390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.031317    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"]
Oct 29 22:58:01.037669 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.037643    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00fa767c-tmmm9"]
Oct 29 22:58:01.097139 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.097111    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-credential-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.097303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.097157    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istiod-ca-cert\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.097303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.097249    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.097303 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.097287    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-podinfo\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.097509 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.097471    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8wm\" (UniqueName: \"kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-kube-api-access-cs8wm\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.097731 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.097706    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.097879 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.097861    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-envoy\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.097976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.097959    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-data\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.098047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.098007    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-token\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.098047 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.098022    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istiod-ca-cert\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.098160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.098046    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-certs\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.098469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.098446    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-certs\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.098569 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.098492    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-credential-socket\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.098722 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.098691    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-data\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.100355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.100295    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-podinfo\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.100471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.100314    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-envoy\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.121210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.121182    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-token\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.121363 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.121249    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8wm\" (UniqueName: \"kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-kube-api-access-cs8wm\") pod \"e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") " pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.209929 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.209901    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:01.267004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.266924    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Oct 29 22:58:01.267391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.267074    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c" podUID="c0d6d3a9-38ae-481f-827b-80d1f3524ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Oct 29 22:58:01.449377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.446741    4860 generic.go:356] "Generic (PLEG): container finished" podID="f7e797a3-5609-40fc-bba4-9e1ab4cdec50" containerID="bd520a3cffa2126811b554290c65a09bb2823d9c440de7c69bec4cc8db1a60c4" exitCode=0
Oct 29 22:58:01.449377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.446772    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5" event={"ID":"f7e797a3-5609-40fc-bba4-9e1ab4cdec50","Type":"ContainerDied","Data":"bd520a3cffa2126811b554290c65a09bb2823d9c440de7c69bec4cc8db1a60c4"}
Oct 29 22:58:01.556305 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:58:01.556273    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb268da_f8b2_4be8_b379_f96e4eb818fc.slice/crio-7f8add58ad8aee947586a07b8db4cc13ef80b84058f57e0e8926360ecd5c0c4d WatchSource:0}: Error finding container 7f8add58ad8aee947586a07b8db4cc13ef80b84058f57e0e8926360ecd5c0c4d: Status 404 returned error can't find the container with id 7f8add58ad8aee947586a07b8db4cc13ef80b84058f57e0e8926360ecd5c0c4d
Oct 29 22:58:01.559883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.559833    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:58:01.560015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.559929    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:58:01.560015 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.559972    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:58:01.561135 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:01.561115    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"]
Oct 29 22:58:02.099562 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.099529    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:02.145014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.144972    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46558b20-00a1-4b08-a62f-889455b0e18f" path="/var/lib/kubelet/pods/46558b20-00a1-4b08-a62f-889455b0e18f/volumes"
Oct 29 22:58:02.266613 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.266579    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Oct 29 22:58:02.266823 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.266632    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c" podUID="c0d6d3a9-38ae-481f-827b-80d1f3524ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Oct 29 22:58:02.458883 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.458841    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt" event={"ID":"ecb268da-f8b2-4be8-b379-f96e4eb818fc","Type":"ContainerStarted","Data":"751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d"}
Oct 29 22:58:02.459283 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.458903    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt" event={"ID":"ecb268da-f8b2-4be8-b379-f96e4eb818fc","Type":"ContainerStarted","Data":"7f8add58ad8aee947586a07b8db4cc13ef80b84058f57e0e8926360ecd5c0c4d"}
Oct 29 22:58:02.509077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.509020    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt" podStartSLOduration=2.5090051239999998 podStartE2EDuration="2.509005124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:02.50682385 +0000 UTC m=+2624.531599983" watchObservedRunningTime="2026-10-29 22:58:02.509005124 +0000 UTC m=+2624.533781254"
Oct 29 22:58:02.744795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.744721    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:58:02.833058 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.833018    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-kube-api-access-5qw6g\" (UniqueName: \"kubernetes.io/projected/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-kube-api-access-5qw6g\") pod \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\" (UID: \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\") "
Oct 29 22:58:02.833441 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.833415    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-openshift-ca\") pod \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\" (UID: \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\") "
Oct 29 22:58:02.833555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.833513    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-xks-ca\") pod \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\" (UID: \"f7e797a3-5609-40fc-bba4-9e1ab4cdec50\") "
Oct 29 22:58:02.835177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.835148    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-openshift-ca" pod "f7e797a3-5609-40fc-bba4-9e1ab4cdec50" (UID: "f7e797a3-5609-40fc-bba4-9e1ab4cdec50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:02.837698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.837668    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-xks-ca" pod "f7e797a3-5609-40fc-bba4-9e1ab4cdec50" (UID: "f7e797a3-5609-40fc-bba4-9e1ab4cdec50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:02.838790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.838765    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-kube-api-access-5qw6g" pod "f7e797a3-5609-40fc-bba4-9e1ab4cdec50" (UID: "f7e797a3-5609-40fc-bba4-9e1ab4cdec50"). InnerVolumeSpecName "kube-api-access-5qw6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:02.936074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.936034    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:02.936074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.936070    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:02.936074 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:02.936082    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qw6g\" (UniqueName: \"kubernetes.io/projected/f7e797a3-5609-40fc-bba4-9e1ab4cdec50-kube-api-access-5qw6g\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:03.214349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:03.213486    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:03.214349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:03.213611    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 29 22:58:03.214349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:03.213658    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt" podUID="ecb268da-f8b2-4be8-b379-f96e4eb818fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 29 22:58:03.265997 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:03.265959    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Oct 29 22:58:03.266167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:03.266006    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c" podUID="c0d6d3a9-38ae-481f-827b-80d1f3524ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Oct 29 22:58:03.524495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:03.524361    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5" event={"ID":"f7e797a3-5609-40fc-bba4-9e1ab4cdec50","Type":"ContainerDied","Data":"9f3fe972c868c19214cc356e38a909984a6f7213c3cf9f93e9183ddd743457d4"}
Oct 29 22:58:03.524495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:03.524424    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f3fe972c868c19214cc356e38a909984a6f7213c3cf9f93e9183ddd743457d4"
Oct 29 22:58:03.525037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:03.524501    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"
Oct 29 22:58:03.630116 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:58:03.630073    4860 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e797a3_5609_40fc_bba4_9e1ab4cdec50.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:58:04.059535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:04.059496    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"]
Oct 29 22:58:04.097746 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:04.097700    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e45f3e7-gw6p8"]
Oct 29 22:58:04.220414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:04.213911    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 29 22:58:04.220414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:04.213971    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt" podUID="ecb268da-f8b2-4be8-b379-f96e4eb818fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 29 22:58:04.265875 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:04.265833    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Oct 29 22:58:04.266081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:04.265906    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c" podUID="c0d6d3a9-38ae-481f-827b-80d1f3524ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Oct 29 22:58:04.320401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:04.319724    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:04.345355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:04.345211    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ba7505-ead5-4b7e-b810-3d6f497b1e14" path="/var/lib/kubelet/pods/80ba7505-ead5-4b7e-b810-3d6f497b1e14/volumes"
Oct 29 22:58:05.210882 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:05.210837    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 29 22:58:05.211289 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:05.210914    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt" podUID="ecb268da-f8b2-4be8-b379-f96e4eb818fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 29 22:58:05.292357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:05.289610    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:58:05.292357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:05.290955    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:58:05.292630 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:05.292612    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 22:58:05.947835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:05.947796    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:06.210954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.210868    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 29 22:58:06.210954 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.210925    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt" podUID="ecb268da-f8b2-4be8-b379-f96e4eb818fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 29 22:58:06.317962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.317928    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"]
Oct 29 22:58:06.318827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.318810    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f7e797a3-5609-40fc-bba4-9e1ab4cdec50" containerName="revoke-keys"
Oct 29 22:58:06.318827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.318827    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7e797a3-5609-40fc-bba4-9e1ab4cdec50" containerName="revoke-keys"
Oct 29 22:58:06.319010 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.319000    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7e797a3-5609-40fc-bba4-9e1ab4cdec50" containerName="revoke-keys"
Oct 29 22:58:06.324421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.324373    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:06.342242 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.342216    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"]
Oct 29 22:58:06.347689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.347666    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:58:06.347689 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.347681    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:06.366298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.366273    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4daa4eb6-8nmqv\" (UID: \"3302b1dc-6642-4da6-ab60-b5c9f156f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:06.366495 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.366472    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4daa4eb6-8nmqv\" (UID: \"3302b1dc-6642-4da6-ab60-b5c9f156f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:06.366545 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.366530    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxx4g\" (UniqueName: \"kubernetes.io/projected/3302b1dc-6642-4da6-ab60-b5c9f156f967-kube-api-access-wxx4g\") pod \"maas-api-revoke-sub-4daa4eb6-8nmqv\" (UID: \"3302b1dc-6642-4da6-ab60-b5c9f156f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:06.492953 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.492852    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4daa4eb6-8nmqv\" (UID: \"3302b1dc-6642-4da6-ab60-b5c9f156f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:06.493145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.492961    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4daa4eb6-8nmqv\" (UID: \"3302b1dc-6642-4da6-ab60-b5c9f156f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:06.493145 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.492993    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxx4g\" (UniqueName: \"kubernetes.io/projected/3302b1dc-6642-4da6-ab60-b5c9f156f967-kube-api-access-wxx4g\") pod \"maas-api-revoke-sub-4daa4eb6-8nmqv\" (UID: \"3302b1dc-6642-4da6-ab60-b5c9f156f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:06.493743 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.493716    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4daa4eb6-8nmqv\" (UID: \"3302b1dc-6642-4da6-ab60-b5c9f156f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:06.495708 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.495682    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4daa4eb6-8nmqv\" (UID: \"3302b1dc-6642-4da6-ab60-b5c9f156f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:06.514469 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.514449    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxx4g\" (UniqueName: \"kubernetes.io/projected/3302b1dc-6642-4da6-ab60-b5c9f156f967-kube-api-access-wxx4g\") pod \"maas-api-revoke-sub-4daa4eb6-8nmqv\" (UID: \"3302b1dc-6642-4da6-ab60-b5c9f156f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:06.641874 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.641829    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:06.759346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.759247    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"]
Oct 29 22:58:06.811230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.811055    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"]
Oct 29 22:58:06.927157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:06.927101    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:07.004996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:07.004970    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"]
Oct 29 22:58:07.052435 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:07.051988    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:07.189355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:07.188453    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:07.224350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:07.211486    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 29 22:58:07.224350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:07.211568    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt" podUID="ecb268da-f8b2-4be8-b379-f96e4eb818fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 29 22:58:07.627981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:07.627939    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:07.670095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:07.670000    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv" event={"ID":"3302b1dc-6642-4da6-ab60-b5c9f156f967","Type":"ContainerStarted","Data":"b660f66c5f04be8b5d59b7e80f285511eb0fbbb02ffb15f894163a1b5c519768"}
Oct 29 22:58:08.031901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:08.031865    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"]
Oct 29 22:58:08.045608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:08.045577    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e65e61-q2ccn"]
Oct 29 22:58:08.137900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:08.137861    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2083e44-42fb-4da8-b978-ee3a76aea574" path="/var/lib/kubelet/pods/d2083e44-42fb-4da8-b978-ee3a76aea574/volumes"
Oct 29 22:58:08.210859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:08.210820    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused" start-of-body=
Oct 29 22:58:08.211014 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:08.210885    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt" podUID="ecb268da-f8b2-4be8-b379-f96e4eb818fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.73:15021/healthz/ready\": dial tcp 10.132.1.73:15021: connect: connection refused"
Oct 29 22:58:08.683044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:08.683012    4860 generic.go:356] "Generic (PLEG): container finished" podID="3302b1dc-6642-4da6-ab60-b5c9f156f967" containerID="2b3e370ff1a0b98ede9c4715cf0f5c96c7313acf5198f226782c933bc54c5283" exitCode=0
Oct 29 22:58:08.683496 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:08.683090    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv" event={"ID":"3302b1dc-6642-4da6-ab60-b5c9f156f967","Type":"ContainerDied","Data":"2b3e370ff1a0b98ede9c4715cf0f5c96c7313acf5198f226782c933bc54c5283"}
Oct 29 22:58:09.276795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:09.276764    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:09.277293 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:09.277267    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:09.278565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:09.278543    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 22:58:09.445515 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:09.445468    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:09.884678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:09.884634    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:10.037903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.037823    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-openshift-ca\") pod \"3302b1dc-6642-4da6-ab60-b5c9f156f967\" (UID: \"3302b1dc-6642-4da6-ab60-b5c9f156f967\") "
Oct 29 22:58:10.037903 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.037884    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3302b1dc-6642-4da6-ab60-b5c9f156f967-kube-api-access-wxx4g\" (UniqueName: \"kubernetes.io/projected/3302b1dc-6642-4da6-ab60-b5c9f156f967-kube-api-access-wxx4g\") pod \"3302b1dc-6642-4da6-ab60-b5c9f156f967\" (UID: \"3302b1dc-6642-4da6-ab60-b5c9f156f967\") "
Oct 29 22:58:10.038265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.038231    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-openshift-ca" pod "3302b1dc-6642-4da6-ab60-b5c9f156f967" (UID: "3302b1dc-6642-4da6-ab60-b5c9f156f967"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:10.038265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.038256    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-xks-ca\") pod \"3302b1dc-6642-4da6-ab60-b5c9f156f967\" (UID: \"3302b1dc-6642-4da6-ab60-b5c9f156f967\") "
Oct 29 22:58:10.038881 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.038859    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:10.040447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.040420    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-xks-ca" pod "3302b1dc-6642-4da6-ab60-b5c9f156f967" (UID: "3302b1dc-6642-4da6-ab60-b5c9f156f967"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:10.040621 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.040597    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3302b1dc-6642-4da6-ab60-b5c9f156f967-kube-api-access-wxx4g" pod "3302b1dc-6642-4da6-ab60-b5c9f156f967" (UID: "3302b1dc-6642-4da6-ab60-b5c9f156f967"). InnerVolumeSpecName "kube-api-access-wxx4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:10.140187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.140158    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3302b1dc-6642-4da6-ab60-b5c9f156f967-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:10.140187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.140184    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxx4g\" (UniqueName: \"kubernetes.io/projected/3302b1dc-6642-4da6-ab60-b5c9f156f967-kube-api-access-wxx4g\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:10.707990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.707957    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"
Oct 29 22:58:10.708256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.707983    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv" event={"ID":"3302b1dc-6642-4da6-ab60-b5c9f156f967","Type":"ContainerDied","Data":"b660f66c5f04be8b5d59b7e80f285511eb0fbbb02ffb15f894163a1b5c519768"}
Oct 29 22:58:10.708256 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.708045    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b660f66c5f04be8b5d59b7e80f285511eb0fbbb02ffb15f894163a1b5c519768"
Oct 29 22:58:10.733030 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.732973    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:10.911945 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:10.911861    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:11.027399 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.027361    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:11.149949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.149918    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:11.219016 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.218981    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"]
Oct 29 22:58:11.219425 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.219215    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49" podUID="d76fbf3e-b816-45d5-876b-284df7a342d2" containerName="payload-processing" containerID="cri-o://6fd06f0ebdf6c3f769c1d6187d13249718daa5c75c42af36844f6ab625ba6ba7" gracePeriod=30
Oct 29 22:58:11.249471 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.249434    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"]
Oct 29 22:58:11.249695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.249668    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq" podUID="d33876b0-ba2e-48d4-87da-24e68486a8fd" containerName="payload-pre-processing" containerID="cri-o://508a866267831e7b38642c62cc675f4f61bf1452ed39d55abea9bdd1abc5d60d" gracePeriod=30
Oct 29 22:58:11.294337 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.294293    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:11.391805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.391546    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"]
Oct 29 22:58:11.401491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.401459    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8c4b02-kn8xg"]
Oct 29 22:58:11.414257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.414227    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:11.772400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.764420    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn" podUID="d57f72e3-b067-4e1e-9295-f6f5f9919255" containerName="payload-processing" containerID="cri-o://61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9" gracePeriod=25
Oct 29 22:58:11.807757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.807481    4860 generic.go:356] "Generic (PLEG): container finished" podID="d33876b0-ba2e-48d4-87da-24e68486a8fd" containerID="508a866267831e7b38642c62cc675f4f61bf1452ed39d55abea9bdd1abc5d60d" exitCode=0
Oct 29 22:58:11.807757 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.807683    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq" event={"ID":"d33876b0-ba2e-48d4-87da-24e68486a8fd","Type":"ContainerDied","Data":"508a866267831e7b38642c62cc675f4f61bf1452ed39d55abea9bdd1abc5d60d"}
Oct 29 22:58:11.818027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.817987    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf" podUID="604c1f26-0ea0-4e6f-ae6c-259716f8a40b" containerName="payload-pre-processing" containerID="cri-o://4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e" gracePeriod=25
Oct 29 22:58:11.825076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.825046    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:11.827372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.827156    4860 generic.go:356] "Generic (PLEG): container finished" podID="d76fbf3e-b816-45d5-876b-284df7a342d2" containerID="6fd06f0ebdf6c3f769c1d6187d13249718daa5c75c42af36844f6ab625ba6ba7" exitCode=0
Oct 29 22:58:11.827555 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.827181    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49" event={"ID":"d76fbf3e-b816-45d5-876b-284df7a342d2","Type":"ContainerDied","Data":"6fd06f0ebdf6c3f769c1d6187d13249718daa5c75c42af36844f6ab625ba6ba7"}
Oct 29 22:58:11.937906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.937859    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:11.959311 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.959284    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"
Oct 29 22:58:11.975717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.975496    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"
Oct 29 22:58:11.984771 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.984715    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"]
Oct 29 22:58:11.995738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:11.995056    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a962d7-v8bd5"]
Oct 29 22:58:12.056706 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.056520    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:12.118572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.095384    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33876b0-ba2e-48d4-87da-24e68486a8fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33876b0-ba2e-48d4-87da-24e68486a8fd-plugins-config-volume\") pod \"d33876b0-ba2e-48d4-87da-24e68486a8fd\" (UID: \"d33876b0-ba2e-48d4-87da-24e68486a8fd\") "
Oct 29 22:58:12.118572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.095456    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76fbf3e-b816-45d5-876b-284df7a342d2-kube-api-access-fj9zf\" (UniqueName: \"kubernetes.io/projected/d76fbf3e-b816-45d5-876b-284df7a342d2-kube-api-access-fj9zf\") pod \"d76fbf3e-b816-45d5-876b-284df7a342d2\" (UID: \"d76fbf3e-b816-45d5-876b-284df7a342d2\") "
Oct 29 22:58:12.118572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.095744    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d76fbf3e-b816-45d5-876b-284df7a342d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d76fbf3e-b816-45d5-876b-284df7a342d2-plugins-config-volume\") pod \"d76fbf3e-b816-45d5-876b-284df7a342d2\" (UID: \"d76fbf3e-b816-45d5-876b-284df7a342d2\") "
Oct 29 22:58:12.118572 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.095837    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33876b0-ba2e-48d4-87da-24e68486a8fd-kube-api-access-4jll7\" (UniqueName: \"kubernetes.io/projected/d33876b0-ba2e-48d4-87da-24e68486a8fd-kube-api-access-4jll7\") pod \"d33876b0-ba2e-48d4-87da-24e68486a8fd\" (UID: \"d33876b0-ba2e-48d4-87da-24e68486a8fd\") "
Oct 29 22:58:12.122484 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.120938    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76fbf3e-b816-45d5-876b-284df7a342d2-plugins-config-volume" pod "d76fbf3e-b816-45d5-876b-284df7a342d2" (UID: "d76fbf3e-b816-45d5-876b-284df7a342d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:12.150004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.139266    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33876b0-ba2e-48d4-87da-24e68486a8fd-plugins-config-volume" pod "d33876b0-ba2e-48d4-87da-24e68486a8fd" (UID: "d33876b0-ba2e-48d4-87da-24e68486a8fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:12.150004 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.146872    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33876b0-ba2e-48d4-87da-24e68486a8fd-kube-api-access-4jll7" pod "d33876b0-ba2e-48d4-87da-24e68486a8fd" (UID: "d33876b0-ba2e-48d4-87da-24e68486a8fd"). InnerVolumeSpecName "kube-api-access-4jll7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:12.176307 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.176275    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:12.183853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.183144    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e91bf163-2b0e-4788-ab6a-05aed58cf339" path="/var/lib/kubelet/pods/e91bf163-2b0e-4788-ab6a-05aed58cf339/volumes"
Oct 29 22:58:12.184906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.184864    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7e797a3-5609-40fc-bba4-9e1ab4cdec50" path="/var/lib/kubelet/pods/f7e797a3-5609-40fc-bba4-9e1ab4cdec50/volumes"
Oct 29 22:58:12.195163 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.195112    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76fbf3e-b816-45d5-876b-284df7a342d2-kube-api-access-fj9zf" pod "d76fbf3e-b816-45d5-876b-284df7a342d2" (UID: "d76fbf3e-b816-45d5-876b-284df7a342d2"). InnerVolumeSpecName "kube-api-access-fj9zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:12.201780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.201687    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jll7\" (UniqueName: \"kubernetes.io/projected/d33876b0-ba2e-48d4-87da-24e68486a8fd-kube-api-access-4jll7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.201780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.201716    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33876b0-ba2e-48d4-87da-24e68486a8fd-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.201780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.201732    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj9zf\" (UniqueName: \"kubernetes.io/projected/d76fbf3e-b816-45d5-876b-284df7a342d2-kube-api-access-fj9zf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.201780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.201747    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d76fbf3e-b816-45d5-876b-284df7a342d2-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.237690 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.237669    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"
Oct 29 22:58:12.362617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.362589    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"
Oct 29 22:58:12.405233 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.405203    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57f72e3-b067-4e1e-9295-f6f5f9919255-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57f72e3-b067-4e1e-9295-f6f5f9919255-plugins-config-volume\") pod \"d57f72e3-b067-4e1e-9295-f6f5f9919255\" (UID: \"d57f72e3-b067-4e1e-9295-f6f5f9919255\") "
Oct 29 22:58:12.405423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.405267    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57f72e3-b067-4e1e-9295-f6f5f9919255-kube-api-access-5hxcn\" (UniqueName: \"kubernetes.io/projected/d57f72e3-b067-4e1e-9295-f6f5f9919255-kube-api-access-5hxcn\") pod \"d57f72e3-b067-4e1e-9295-f6f5f9919255\" (UID: \"d57f72e3-b067-4e1e-9295-f6f5f9919255\") "
Oct 29 22:58:12.405681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.405652    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57f72e3-b067-4e1e-9295-f6f5f9919255-plugins-config-volume" pod "d57f72e3-b067-4e1e-9295-f6f5f9919255" (UID: "d57f72e3-b067-4e1e-9295-f6f5f9919255"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:12.405972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.405954    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57f72e3-b067-4e1e-9295-f6f5f9919255-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.407685 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.407642    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57f72e3-b067-4e1e-9295-f6f5f9919255-kube-api-access-5hxcn" pod "d57f72e3-b067-4e1e-9295-f6f5f9919255" (UID: "d57f72e3-b067-4e1e-9295-f6f5f9919255"). InnerVolumeSpecName "kube-api-access-5hxcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:12.469056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.469024    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:12.507016 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.506991    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-kube-api-access-bkwhm\" (UniqueName: \"kubernetes.io/projected/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-kube-api-access-bkwhm\") pod \"604c1f26-0ea0-4e6f-ae6c-259716f8a40b\" (UID: \"604c1f26-0ea0-4e6f-ae6c-259716f8a40b\") "
Oct 29 22:58:12.507126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.507048    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-plugins-config-volume\") pod \"604c1f26-0ea0-4e6f-ae6c-259716f8a40b\" (UID: \"604c1f26-0ea0-4e6f-ae6c-259716f8a40b\") "
Oct 29 22:58:12.507521 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.507498    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-plugins-config-volume" pod "604c1f26-0ea0-4e6f-ae6c-259716f8a40b" (UID: "604c1f26-0ea0-4e6f-ae6c-259716f8a40b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:12.507866 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.507851    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hxcn\" (UniqueName: \"kubernetes.io/projected/d57f72e3-b067-4e1e-9295-f6f5f9919255-kube-api-access-5hxcn\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.507906 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.507870    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.509138 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.509121    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-kube-api-access-bkwhm" pod "604c1f26-0ea0-4e6f-ae6c-259716f8a40b" (UID: "604c1f26-0ea0-4e6f-ae6c-259716f8a40b"). InnerVolumeSpecName "kube-api-access-bkwhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:12.609343 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.609271    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkwhm\" (UniqueName: \"kubernetes.io/projected/604c1f26-0ea0-4e6f-ae6c-259716f8a40b-kube-api-access-bkwhm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.840534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.840500    4860 generic.go:356] "Generic (PLEG): container finished" podID="d57f72e3-b067-4e1e-9295-f6f5f9919255" containerID="61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9" exitCode=0
Oct 29 22:58:12.840701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.840549    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"
Oct 29 22:58:12.840701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.840584    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn" event={"ID":"d57f72e3-b067-4e1e-9295-f6f5f9919255","Type":"ContainerDied","Data":"61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9"}
Oct 29 22:58:12.840701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.840642    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn" event={"ID":"d57f72e3-b067-4e1e-9295-f6f5f9919255","Type":"ContainerDied","Data":"ffb2633afe6dca09a9cd022de952af9d50f266c284729810a54ae8d8c8d93fc6"}
Oct 29 22:58:12.840701 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.840669    4860 scope.go:122] "RemoveContainer" containerID="61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9"
Oct 29 22:58:12.842403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.842371    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq" event={"ID":"d33876b0-ba2e-48d4-87da-24e68486a8fd","Type":"ContainerDied","Data":"d65e806ce935b53025020b2e814dab4a91cad76296e46bc2c15d725445e5c0ef"}
Oct 29 22:58:12.842568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.842399    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"
Oct 29 22:58:12.844254 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.844232    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"
Oct 29 22:58:12.844430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.844263    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49" event={"ID":"d76fbf3e-b816-45d5-876b-284df7a342d2","Type":"ContainerDied","Data":"fe9fbec496c339b332b98d7b06263bd47a2a8d38fb3f21f05f16a32ba975d8bf"}
Oct 29 22:58:12.846453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.846429    4860 generic.go:356] "Generic (PLEG): container finished" podID="604c1f26-0ea0-4e6f-ae6c-259716f8a40b" containerID="4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e" exitCode=0
Oct 29 22:58:12.846578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.846464    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"
Oct 29 22:58:12.846578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.846474    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf" event={"ID":"604c1f26-0ea0-4e6f-ae6c-259716f8a40b","Type":"ContainerDied","Data":"4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e"}
Oct 29 22:58:12.846578 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.846511    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf" event={"ID":"604c1f26-0ea0-4e6f-ae6c-259716f8a40b","Type":"ContainerDied","Data":"0f036ee21988563e8117d35d71949f66d98414943e0e42b10313ff94e266cf5a"}
Oct 29 22:58:12.855315 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.855291    4860 scope.go:122] "RemoveContainer" containerID="61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9"
Oct 29 22:58:12.856188 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:58:12.855813    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9\": container with ID starting with 61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9 not found: ID does not exist" containerID="61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9"
Oct 29 22:58:12.856188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.856160    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9"} err="failed to get container status \"61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9\": rpc error: code = NotFound desc = could not find container \"61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9\": container with ID starting with 61905f4ec1a384aa99f81e84a6136f6b4205aadf523603c5eccf87d18a5d2de9 not found: ID does not exist"
Oct 29 22:58:12.856188 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.856180    4860 scope.go:122] "RemoveContainer" containerID="508a866267831e7b38642c62cc675f4f61bf1452ed39d55abea9bdd1abc5d60d"
Oct 29 22:58:12.873723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.873704    4860 scope.go:122] "RemoveContainer" containerID="6fd06f0ebdf6c3f769c1d6187d13249718daa5c75c42af36844f6ab625ba6ba7"
Oct 29 22:58:12.887261 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.887240    4860 scope.go:122] "RemoveContainer" containerID="4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e"
Oct 29 22:58:12.897653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.897634    4860 scope.go:122] "RemoveContainer" containerID="4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e"
Oct 29 22:58:12.897916 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:58:12.897896    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e\": container with ID starting with 4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e not found: ID does not exist" containerID="4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e"
Oct 29 22:58:12.897995 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.897922    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e"} err="failed to get container status \"4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e\": rpc error: code = NotFound desc = could not find container \"4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e\": container with ID starting with 4c7b38e38ca93790d17b4afa1706720c88bd2580aaa16b1204eb4b9e4111b84e not found: ID does not exist"
Oct 29 22:58:12.932843 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.932823    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"]
Oct 29 22:58:12.945849 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.945828    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a962d7-688dd94bc-xptcq"]
Oct 29 22:58:12.972602 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.972578    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"]
Oct 29 22:58:12.977928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:12.977905    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85862e-5759d9df58-t4vrn"]
Oct 29 22:58:13.014048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:13.014022    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"]
Oct 29 22:58:13.019149 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:13.019125    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-597bbb855-5jrdf"]
Oct 29 22:58:13.054078 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:13.054053    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"]
Oct 29 22:58:13.059378 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:13.059357    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a962d7-55d9d5c8c4-dfl49"]
Oct 29 22:58:13.875687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:13.875654    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:14.117841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:14.117811    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604c1f26-0ea0-4e6f-ae6c-259716f8a40b" path="/var/lib/kubelet/pods/604c1f26-0ea0-4e6f-ae6c-259716f8a40b/volumes"
Oct 29 22:58:14.118357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:14.118341    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33876b0-ba2e-48d4-87da-24e68486a8fd" path="/var/lib/kubelet/pods/d33876b0-ba2e-48d4-87da-24e68486a8fd/volumes"
Oct 29 22:58:14.118808 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:14.118793    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57f72e3-b067-4e1e-9295-f6f5f9919255" path="/var/lib/kubelet/pods/d57f72e3-b067-4e1e-9295-f6f5f9919255/volumes"
Oct 29 22:58:14.119251 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:14.119236    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76fbf3e-b816-45d5-876b-284df7a342d2" path="/var/lib/kubelet/pods/d76fbf3e-b816-45d5-876b-284df7a342d2/volumes"
Oct 29 22:58:14.697260 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:14.697227    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:15.291382 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:15.291318    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:15.541926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:15.541823    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:16.653949 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.653916    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:16.858580 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.858547    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:16.873540 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.873474    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"]
Oct 29 22:58:16.874853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.874812    4860 cpu_manager.go:414] "containerMap: removing container" podUID="604c1f26-0ea0-4e6f-ae6c-259716f8a40b" containerName="payload-pre-processing"
Oct 29 22:58:16.874853 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.874853    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604c1f26-0ea0-4e6f-ae6c-259716f8a40b" containerName="payload-pre-processing"
Oct 29 22:58:16.875066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.874928    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d57f72e3-b067-4e1e-9295-f6f5f9919255" containerName="payload-processing"
Oct 29 22:58:16.875066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.874938    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57f72e3-b067-4e1e-9295-f6f5f9919255" containerName="payload-processing"
Oct 29 22:58:16.875066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.874968    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d76fbf3e-b816-45d5-876b-284df7a342d2" containerName="payload-processing"
Oct 29 22:58:16.875066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.874975    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76fbf3e-b816-45d5-876b-284df7a342d2" containerName="payload-processing"
Oct 29 22:58:16.875066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.874985    4860 cpu_manager.go:414] "containerMap: removing container" podUID="3302b1dc-6642-4da6-ab60-b5c9f156f967" containerName="revoke-keys"
Oct 29 22:58:16.875066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.874992    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3302b1dc-6642-4da6-ab60-b5c9f156f967" containerName="revoke-keys"
Oct 29 22:58:16.875066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.875004    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d33876b0-ba2e-48d4-87da-24e68486a8fd" containerName="payload-pre-processing"
Oct 29 22:58:16.875066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.875011    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33876b0-ba2e-48d4-87da-24e68486a8fd" containerName="payload-pre-processing"
Oct 29 22:58:16.875551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.875239    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="604c1f26-0ea0-4e6f-ae6c-259716f8a40b" containerName="payload-pre-processing"
Oct 29 22:58:16.875551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.875266    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57f72e3-b067-4e1e-9295-f6f5f9919255" containerName="payload-processing"
Oct 29 22:58:16.875551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.875281    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76fbf3e-b816-45d5-876b-284df7a342d2" containerName="payload-processing"
Oct 29 22:58:16.875551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.875290    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="3302b1dc-6642-4da6-ab60-b5c9f156f967" containerName="revoke-keys"
Oct 29 22:58:16.875551 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.875300    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33876b0-ba2e-48d4-87da-24e68486a8fd" containerName="payload-pre-processing"
Oct 29 22:58:16.893244 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.893212    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:16.944624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.939095    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:16.944624 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.939272    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:58:16.946103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:16.946074    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"]
Oct 29 22:58:17.054347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.039950    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eba808e-qj8tm\" (UID: \"b27c2001-f753-4594-835a-c75f45db3ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:17.054347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.040095    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eba808e-qj8tm\" (UID: \"b27c2001-f753-4594-835a-c75f45db3ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:17.054347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.040142    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g755g\" (UniqueName: \"kubernetes.io/projected/b27c2001-f753-4594-835a-c75f45db3ed1-kube-api-access-g755g\") pod \"maas-api-revoke-sub-9eba808e-qj8tm\" (UID: \"b27c2001-f753-4594-835a-c75f45db3ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:17.210346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.207994    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g755g\" (UniqueName: \"kubernetes.io/projected/b27c2001-f753-4594-835a-c75f45db3ed1-kube-api-access-g755g\") pod \"maas-api-revoke-sub-9eba808e-qj8tm\" (UID: \"b27c2001-f753-4594-835a-c75f45db3ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:17.210346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.208259    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eba808e-qj8tm\" (UID: \"b27c2001-f753-4594-835a-c75f45db3ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:17.210346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.208390    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eba808e-qj8tm\" (UID: \"b27c2001-f753-4594-835a-c75f45db3ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:17.227354 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.224040    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9eba808e-qj8tm\" (UID: \"b27c2001-f753-4594-835a-c75f45db3ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:17.233936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.233885    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9eba808e-qj8tm\" (UID: \"b27c2001-f753-4594-835a-c75f45db3ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:17.258503 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.258422    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g755g\" (UniqueName: \"kubernetes.io/projected/b27c2001-f753-4594-835a-c75f45db3ed1-kube-api-access-g755g\") pod \"maas-api-revoke-sub-9eba808e-qj8tm\" (UID: \"b27c2001-f753-4594-835a-c75f45db3ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:17.552147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.552059    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:17.759859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.759821    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:17.887752 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:58:17.887173    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27c2001_f753_4594_835a_c75f45db3ed1.slice/crio-4a1cf8b24e38c96d0740fa920d0387a7e88c61f40df421cf6ab0194762e80158 WatchSource:0}: Error finding container 4a1cf8b24e38c96d0740fa920d0387a7e88c61f40df421cf6ab0194762e80158: Status 404 returned error can't find the container with id 4a1cf8b24e38c96d0740fa920d0387a7e88c61f40df421cf6ab0194762e80158
Oct 29 22:58:17.894796 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.894753    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"]
Oct 29 22:58:17.929533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:17.929509    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm" event={"ID":"b27c2001-f753-4594-835a-c75f45db3ed1","Type":"ContainerStarted","Data":"4a1cf8b24e38c96d0740fa920d0387a7e88c61f40df421cf6ab0194762e80158"}
Oct 29 22:58:18.058412 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:58:18.058375    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:58:18.648537 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:18.648463    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:18.775666 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:18.775631    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:18.902371 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:18.902275    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"]
Oct 29 22:58:18.902575 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:18.902547    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" podUID="d5299767-caa1-4246-b4c1-9217642edfab" containerName="istio-proxy" containerID="cri-o://e0341e11e57acb675c60d577c5e22ed67ae7d400467112936b88f746eaa30ee2" gracePeriod=30
Oct 29 22:58:18.941527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:18.941498    4860 generic.go:356] "Generic (PLEG): container finished" podID="b27c2001-f753-4594-835a-c75f45db3ed1" containerID="2edc608078c6c34526400eeec3a1ae5ba4da5b0f7032aeee49411373a71e3340" exitCode=0
Oct 29 22:58:18.941650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:18.941588    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm" event={"ID":"b27c2001-f753-4594-835a-c75f45db3ed1","Type":"ContainerDied","Data":"2edc608078c6c34526400eeec3a1ae5ba4da5b0f7032aeee49411373a71e3340"}
Oct 29 22:58:18.949067 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:18.949044    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:19.140620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:19.140586    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:19.163476 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:19.149844    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"]
Oct 29 22:58:19.222854 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:19.222812    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0245f65-sgj2w"]
Oct 29 22:58:19.288440 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:19.288394    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:19.446420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:19.446387    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:19.576241 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:19.576082    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:19.917758 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:19.917679    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"]
Oct 29 22:58:19.922724 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:19.922704    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:19.943846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:19.943813    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"]
Oct 29 22:58:19.973115 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:19.973077    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n6xw\" (UniqueName: \"kubernetes.io/projected/9e1972ec-5c66-4862-a66c-b623403a4414-kube-api-access-7n6xw\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l\" (UID: \"9e1972ec-5c66-4862-a66c-b623403a4414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:19.973406 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:19.973375    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l\" (UID: \"9e1972ec-5c66-4862-a66c-b623403a4414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:19.973619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:19.973524    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l\" (UID: \"9e1972ec-5c66-4862-a66c-b623403a4414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:20.075101 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.075057    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l\" (UID: \"9e1972ec-5c66-4862-a66c-b623403a4414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:20.075290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.075197    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l\" (UID: \"9e1972ec-5c66-4862-a66c-b623403a4414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:20.075421 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.075369    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n6xw\" (UniqueName: \"kubernetes.io/projected/9e1972ec-5c66-4862-a66c-b623403a4414-kube-api-access-7n6xw\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l\" (UID: \"9e1972ec-5c66-4862-a66c-b623403a4414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:20.076048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.076009    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l\" (UID: \"9e1972ec-5c66-4862-a66c-b623403a4414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:20.078604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.078575    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l\" (UID: \"9e1972ec-5c66-4862-a66c-b623403a4414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:20.098229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.098207    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n6xw\" (UniqueName: \"kubernetes.io/projected/9e1972ec-5c66-4862-a66c-b623403a4414-kube-api-access-7n6xw\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l\" (UID: \"9e1972ec-5c66-4862-a66c-b623403a4414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:20.118985 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.118959    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1bbd5d-f28c-4f80-ace9-52820f32f954" path="/var/lib/kubelet/pods/8b1bbd5d-f28c-4f80-ace9-52820f32f954/volumes"
Oct 29 22:58:20.148587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.148566    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:20.176342 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.176272    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-openshift-ca\") pod \"b27c2001-f753-4594-835a-c75f45db3ed1\" (UID: \"b27c2001-f753-4594-835a-c75f45db3ed1\") "
Oct 29 22:58:20.176463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.176418    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b27c2001-f753-4594-835a-c75f45db3ed1-kube-api-access-g755g\" (UniqueName: \"kubernetes.io/projected/b27c2001-f753-4594-835a-c75f45db3ed1-kube-api-access-g755g\") pod \"b27c2001-f753-4594-835a-c75f45db3ed1\" (UID: \"b27c2001-f753-4594-835a-c75f45db3ed1\") "
Oct 29 22:58:20.176516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.176484    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-xks-ca\") pod \"b27c2001-f753-4594-835a-c75f45db3ed1\" (UID: \"b27c2001-f753-4594-835a-c75f45db3ed1\") "
Oct 29 22:58:20.176635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.176614    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-openshift-ca" pod "b27c2001-f753-4594-835a-c75f45db3ed1" (UID: "b27c2001-f753-4594-835a-c75f45db3ed1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:20.177143 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.177122    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:20.178631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.178606    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-xks-ca" pod "b27c2001-f753-4594-835a-c75f45db3ed1" (UID: "b27c2001-f753-4594-835a-c75f45db3ed1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:20.178723 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.178694    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27c2001-f753-4594-835a-c75f45db3ed1-kube-api-access-g755g" pod "b27c2001-f753-4594-835a-c75f45db3ed1" (UID: "b27c2001-f753-4594-835a-c75f45db3ed1"). InnerVolumeSpecName "kube-api-access-g755g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:20.241843 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.241814    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:20.279272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.279241    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g755g\" (UniqueName: \"kubernetes.io/projected/b27c2001-f753-4594-835a-c75f45db3ed1-kube-api-access-g755g\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:20.279272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.279272    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27c2001-f753-4594-835a-c75f45db3ed1-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:20.548007 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:58:20.547974    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1972ec_5c66_4862_a66c_b623403a4414.slice/crio-3b5917d345505ebb62ab77f6e1b040b0420db0956ae610e8803a94f7b6370a40 WatchSource:0}: Error finding container 3b5917d345505ebb62ab77f6e1b040b0420db0956ae610e8803a94f7b6370a40: Status 404 returned error can't find the container with id 3b5917d345505ebb62ab77f6e1b040b0420db0956ae610e8803a94f7b6370a40
Oct 29 22:58:20.553165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.553139    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"]
Oct 29 22:58:20.967432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.967402    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"
Oct 29 22:58:20.967432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.967420    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm" event={"ID":"b27c2001-f753-4594-835a-c75f45db3ed1","Type":"ContainerDied","Data":"4a1cf8b24e38c96d0740fa920d0387a7e88c61f40df421cf6ab0194762e80158"}
Oct 29 22:58:20.967952 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.967467    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a1cf8b24e38c96d0740fa920d0387a7e88c61f40df421cf6ab0194762e80158"
Oct 29 22:58:20.969105 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:20.969080    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l" event={"ID":"9e1972ec-5c66-4862-a66c-b623403a4414","Type":"ContainerStarted","Data":"3b5917d345505ebb62ab77f6e1b040b0420db0956ae610e8803a94f7b6370a40"}
Oct 29 22:58:21.440625 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:21.440594    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:21.778264 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:21.778179    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:21.980429 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:21.980393    4860 generic.go:356] "Generic (PLEG): container finished" podID="9e1972ec-5c66-4862-a66c-b623403a4414" containerID="bd136255f162d32b41eb38f1d180b3900d5f12ecccdeb73345e43bb23d989f51" exitCode=0
Oct 29 22:58:21.980918 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:21.980473    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l" event={"ID":"9e1972ec-5c66-4862-a66c-b623403a4414","Type":"ContainerDied","Data":"bd136255f162d32b41eb38f1d180b3900d5f12ecccdeb73345e43bb23d989f51"}
Oct 29 22:58:23.009574 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.009536    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:23.280487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.280464    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:23.335416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.335379    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1972ec-5c66-4862-a66c-b623403a4414-kube-api-access-7n6xw\" (UniqueName: \"kubernetes.io/projected/9e1972ec-5c66-4862-a66c-b623403a4414-kube-api-access-7n6xw\") pod \"9e1972ec-5c66-4862-a66c-b623403a4414\" (UID: \"9e1972ec-5c66-4862-a66c-b623403a4414\") "
Oct 29 22:58:23.335629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.335463    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-xks-ca\") pod \"9e1972ec-5c66-4862-a66c-b623403a4414\" (UID: \"9e1972ec-5c66-4862-a66c-b623403a4414\") "
Oct 29 22:58:23.335629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.335528    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-openshift-ca\") pod \"9e1972ec-5c66-4862-a66c-b623403a4414\" (UID: \"9e1972ec-5c66-4862-a66c-b623403a4414\") "
Oct 29 22:58:23.336820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.336789    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-openshift-ca" pod "9e1972ec-5c66-4862-a66c-b623403a4414" (UID: "9e1972ec-5c66-4862-a66c-b623403a4414"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:23.338742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.338716    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-xks-ca" pod "9e1972ec-5c66-4862-a66c-b623403a4414" (UID: "9e1972ec-5c66-4862-a66c-b623403a4414"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:23.339216 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.339186    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1972ec-5c66-4862-a66c-b623403a4414-kube-api-access-7n6xw" pod "9e1972ec-5c66-4862-a66c-b623403a4414" (UID: "9e1972ec-5c66-4862-a66c-b623403a4414"). InnerVolumeSpecName "kube-api-access-7n6xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:23.438581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.438555    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n6xw\" (UniqueName: \"kubernetes.io/projected/9e1972ec-5c66-4862-a66c-b623403a4414-kube-api-access-7n6xw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:23.438581 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.438581    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:23.438745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.438593    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e1972ec-5c66-4862-a66c-b623403a4414-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:23.921761 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.921726    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"]
Oct 29 22:58:23.924192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.924164    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b27c2001-f753-4594-835a-c75f45db3ed1" containerName="revoke-keys"
Oct 29 22:58:23.924192 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.924191    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27c2001-f753-4594-835a-c75f45db3ed1" containerName="revoke-keys"
Oct 29 22:58:23.924450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.924222    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9e1972ec-5c66-4862-a66c-b623403a4414" containerName="revoke-keys"
Oct 29 22:58:23.924450 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.924230    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1972ec-5c66-4862-a66c-b623403a4414" containerName="revoke-keys"
Oct 29 22:58:23.926005 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.925355    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27c2001-f753-4594-835a-c75f45db3ed1" containerName="revoke-keys"
Oct 29 22:58:23.926005 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.925391    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1972ec-5c66-4862-a66c-b623403a4414" containerName="revoke-keys"
Oct 29 22:58:23.940539 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.940503    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:23.950986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:23.950946    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"]
Oct 29 22:58:24.018610 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.018574    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l" event={"ID":"9e1972ec-5c66-4862-a66c-b623403a4414","Type":"ContainerDied","Data":"3b5917d345505ebb62ab77f6e1b040b0420db0956ae610e8803a94f7b6370a40"}
Oct 29 22:58:24.019076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.018636    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b5917d345505ebb62ab77f6e1b040b0420db0956ae610e8803a94f7b6370a40"
Oct 29 22:58:24.019076 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.018704    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"
Oct 29 22:58:24.027776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.027732    4860 generic.go:356] "Generic (PLEG): container finished" podID="d5299767-caa1-4246-b4c1-9217642edfab" containerID="e0341e11e57acb675c60d577c5e22ed67ae7d400467112936b88f746eaa30ee2" exitCode=0
Oct 29 22:58:24.027993 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.027969    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" event={"ID":"d5299767-caa1-4246-b4c1-9217642edfab","Type":"ContainerDied","Data":"e0341e11e57acb675c60d577c5e22ed67ae7d400467112936b88f746eaa30ee2"}
Oct 29 22:58:24.028126 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.028114    4860 scope.go:122] "RemoveContainer" containerID="992f3625e72cc437f6d1d36a41ca8cada4b50ee622d49920f870b1d85fb341ba"
Oct 29 22:58:24.048550 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.048522    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82\" (UID: \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:24.048688 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.048667    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghjs\" (UniqueName: \"kubernetes.io/projected/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-kube-api-access-mghjs\") pod \"maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82\" (UID: \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:24.048778 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.048763    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82\" (UID: \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:24.150390 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.150317    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghjs\" (UniqueName: \"kubernetes.io/projected/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-kube-api-access-mghjs\") pod \"maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82\" (UID: \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:24.150560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.150480    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82\" (UID: \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:24.151205 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.150788    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82\" (UID: \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:24.151205 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.151161    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82\" (UID: \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:24.152294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.152272    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:58:24.153053 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.153036    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82\" (UID: \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:24.171185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.171136    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghjs\" (UniqueName: \"kubernetes.io/projected/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-kube-api-access-mghjs\") pod \"maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82\" (UID: \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:24.252652 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.252613    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-socket" pod "d5299767-caa1-4246-b4c1-9217642edfab" (UID: "d5299767-caa1-4246-b4c1-9217642edfab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:24.252825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.252316    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-socket\") pod \"d5299767-caa1-4246-b4c1-9217642edfab\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") "
Oct 29 22:58:24.252825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.252787    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-data\") pod \"d5299767-caa1-4246-b4c1-9217642edfab\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") "
Oct 29 22:58:24.253479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.253452    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-data" pod "d5299767-caa1-4246-b4c1-9217642edfab" (UID: "d5299767-caa1-4246-b4c1-9217642edfab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:24.253594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.253496    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-kube-api-access-ft6fp\" (UniqueName: \"kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-kube-api-access-ft6fp\") pod \"d5299767-caa1-4246-b4c1-9217642edfab\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") "
Oct 29 22:58:24.253594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.253568    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5299767-caa1-4246-b4c1-9217642edfab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5299767-caa1-4246-b4c1-9217642edfab-istio-podinfo\") pod \"d5299767-caa1-4246-b4c1-9217642edfab\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") "
Oct 29 22:58:24.253780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.253594    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5299767-caa1-4246-b4c1-9217642edfab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5299767-caa1-4246-b4c1-9217642edfab-istiod-ca-cert\") pod \"d5299767-caa1-4246-b4c1-9217642edfab\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") "
Oct 29 22:58:24.254571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.253778    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-certs\") pod \"d5299767-caa1-4246-b4c1-9217642edfab\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") "
Oct 29 22:58:24.254571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.253828    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-envoy\") pod \"d5299767-caa1-4246-b4c1-9217642edfab\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") "
Oct 29 22:58:24.254571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.253856    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-istio-token\" (UniqueName: \"kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-istio-token\") pod \"d5299767-caa1-4246-b4c1-9217642edfab\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") "
Oct 29 22:58:24.254571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.253989    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-credential-socket\") pod \"d5299767-caa1-4246-b4c1-9217642edfab\" (UID: \"d5299767-caa1-4246-b4c1-9217642edfab\") "
Oct 29 22:58:24.254571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.254102    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-certs" pod "d5299767-caa1-4246-b4c1-9217642edfab" (UID: "d5299767-caa1-4246-b4c1-9217642edfab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:24.254571 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.254440    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5299767-caa1-4246-b4c1-9217642edfab-istiod-ca-cert" pod "d5299767-caa1-4246-b4c1-9217642edfab" (UID: "d5299767-caa1-4246-b4c1-9217642edfab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:24.256662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.256639    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-credential-socket" pod "d5299767-caa1-4246-b4c1-9217642edfab" (UID: "d5299767-caa1-4246-b4c1-9217642edfab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:24.258768 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.258747    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5299767-caa1-4246-b4c1-9217642edfab-istio-podinfo" pod "d5299767-caa1-4246-b4c1-9217642edfab" (UID: "d5299767-caa1-4246-b4c1-9217642edfab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:58:24.259012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.258984    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:24.259132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.259019    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:24.259132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.259032    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:24.259132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.259045    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5299767-caa1-4246-b4c1-9217642edfab-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:24.259132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.259058    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:24.259528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.259499    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-envoy" pod "d5299767-caa1-4246-b4c1-9217642edfab" (UID: "d5299767-caa1-4246-b4c1-9217642edfab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:58:24.259916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.259889    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-kube-api-access-ft6fp" pod "d5299767-caa1-4246-b4c1-9217642edfab" (UID: "d5299767-caa1-4246-b4c1-9217642edfab"). InnerVolumeSpecName "kube-api-access-ft6fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:24.261698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.261673    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-istio-token" pod "d5299767-caa1-4246-b4c1-9217642edfab" (UID: "d5299767-caa1-4246-b4c1-9217642edfab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:24.307648 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.307611    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:24.360820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.360790    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft6fp\" (UniqueName: \"kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-kube-api-access-ft6fp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:24.360820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.360819    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5299767-caa1-4246-b4c1-9217642edfab-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:24.360984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.360830    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5299767-caa1-4246-b4c1-9217642edfab-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:24.360984 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.360839    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5299767-caa1-4246-b4c1-9217642edfab-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:24.480212 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.480178    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"]
Oct 29 22:58:24.481093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.481072    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d5299767-caa1-4246-b4c1-9217642edfab" containerName="istio-proxy"
Oct 29 22:58:24.481093 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.481091    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5299767-caa1-4246-b4c1-9217642edfab" containerName="istio-proxy"
Oct 29 22:58:24.481249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.481136    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d5299767-caa1-4246-b4c1-9217642edfab" containerName="istio-proxy"
Oct 29 22:58:24.481249 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.481141    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5299767-caa1-4246-b4c1-9217642edfab" containerName="istio-proxy"
Oct 29 22:58:24.481366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.481298    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5299767-caa1-4246-b4c1-9217642edfab" containerName="istio-proxy"
Oct 29 22:58:24.481366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.481315    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5299767-caa1-4246-b4c1-9217642edfab" containerName="istio-proxy"
Oct 29 22:58:24.487560 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.487542    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.506443 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.506414    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-83f120-serving-cert\""
Oct 29 22:58:24.508380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.508355    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"]
Oct 29 22:58:24.565110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.565077    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be999b8c-3f77-451a-9d23-be1db342ce44-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.565262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.565220    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7jz\" (UniqueName: \"kubernetes.io/projected/be999b8c-3f77-451a-9d23-be1db342ce44-kube-api-access-vc7jz\") pod \"maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.565402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.565289    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-metrics-certs\") pod \"maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.565402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.565320    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-maas-api-tls\") pod \"maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.576404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.576371    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:24.618589 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:58:24.618563    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0701f3d8_5c09_4cca_b5e2_c21c3fa16c4a.slice/crio-47a8c419c2832d4047cbf9e3bd553a3503fb73d7ca2d17df17db366f1d743f3b WatchSource:0}: Error finding container 47a8c419c2832d4047cbf9e3bd553a3503fb73d7ca2d17df17db366f1d743f3b: Status 404 returned error can't find the container with id 47a8c419c2832d4047cbf9e3bd553a3503fb73d7ca2d17df17db366f1d743f3b
Oct 29 22:58:24.622638 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.622616    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"]
Oct 29 22:58:24.682650 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.682599    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be999b8c-3f77-451a-9d23-be1db342ce44-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.682858 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.682789    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7jz\" (UniqueName: \"kubernetes.io/projected/be999b8c-3f77-451a-9d23-be1db342ce44-kube-api-access-vc7jz\") pod \"maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.682926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.682881    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-metrics-certs\") pod \"maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.682926 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.682918    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-maas-api-tls\") pod \"maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.689789 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.689746    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be999b8c-3f77-451a-9d23-be1db342ce44-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.697236 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.697196    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:24.771566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.770951    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-maas-api-tls\") pod \"maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.772946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.772488    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7jz\" (UniqueName: \"kubernetes.io/projected/be999b8c-3f77-451a-9d23-be1db342ce44-kube-api-access-vc7jz\") pod \"maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.772946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.772906    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-metrics-certs\") pod \"maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:24.808411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:24.808381    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:25.039273 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:25.039187    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82" event={"ID":"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a","Type":"ContainerStarted","Data":"47a8c419c2832d4047cbf9e3bd553a3503fb73d7ca2d17df17db366f1d743f3b"}
Oct 29 22:58:25.040847 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:25.040819    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"
Oct 29 22:58:25.040989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:25.040848    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh" event={"ID":"d5299767-caa1-4246-b4c1-9217642edfab","Type":"ContainerDied","Data":"0e7d7aaf73e88403d0e4788c89f07990f4d5d9534fd395b011ad939fe634f7ff"}
Oct 29 22:58:25.040989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:25.040899    4860 scope.go:122] "RemoveContainer" containerID="e0341e11e57acb675c60d577c5e22ed67ae7d400467112936b88f746eaa30ee2"
Oct 29 22:58:25.101130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:25.101101    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"]
Oct 29 22:58:25.109490 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:25.109466    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8c4b02-openshift-default-679994b96f-7wssh"]
Oct 29 22:58:25.121751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:25.121727    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"]
Oct 29 22:58:26.055742 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:26.055669    4860 generic.go:356] "Generic (PLEG): container finished" podID="0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a" containerID="a16dd42fdd30462f4d8fe0153772104d4a28d270085105fb5b281cebfa2d5f19" exitCode=0
Oct 29 22:58:26.056245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:26.055754    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82" event={"ID":"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a","Type":"ContainerDied","Data":"a16dd42fdd30462f4d8fe0153772104d4a28d270085105fb5b281cebfa2d5f19"}
Oct 29 22:58:26.059183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:26.059157    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c" event={"ID":"be999b8c-3f77-451a-9d23-be1db342ce44","Type":"ContainerStarted","Data":"a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1"}
Oct 29 22:58:26.059319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:26.059206    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c" event={"ID":"be999b8c-3f77-451a-9d23-be1db342ce44","Type":"ContainerStarted","Data":"ff83f37f725db575e739ea67bcfbd1514232fff0d7c6ce787c528aa7e5851ec9"}
Oct 29 22:58:26.059319 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:26.059254    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:26.116438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:26.116408    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5299767-caa1-4246-b4c1-9217642edfab" path="/var/lib/kubelet/pods/d5299767-caa1-4246-b4c1-9217642edfab/volumes"
Oct 29 22:58:26.153453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:26.153388    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c" podStartSLOduration=1.854074875 podStartE2EDuration="2.153367689s" totalImagesPullingTime="299.292814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:26.142652057 +0000 UTC m=+2648.167428210" watchObservedRunningTime="2026-10-29 22:58:26.153367689 +0000 UTC m=+2648.178143823"
Oct 29 22:58:26.440222 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:26.440194    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:27.250243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:27.250218    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:27.378525 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:27.378447    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-openshift-ca\") pod \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\" (UID: \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\") "
Oct 29 22:58:27.378702 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:27.378591    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-kube-api-access-mghjs\" (UniqueName: \"kubernetes.io/projected/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-kube-api-access-mghjs\") pod \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\" (UID: \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\") "
Oct 29 22:58:27.378803 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:27.378787    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-xks-ca\") pod \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\" (UID: \"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a\") "
Oct 29 22:58:27.378892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:27.378814    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-openshift-ca" pod "0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a" (UID: "0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:27.379411 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:27.379384    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:27.380981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:27.380953    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-kube-api-access-mghjs" pod "0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a" (UID: "0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a"). InnerVolumeSpecName "kube-api-access-mghjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:27.380981 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:27.380959    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-xks-ca" pod "0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a" (UID: "0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:27.481502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:27.481471    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mghjs\" (UniqueName: \"kubernetes.io/projected/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-kube-api-access-mghjs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:27.481502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:27.481498    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:27.673052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:27.672927    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:28.184730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:28.184698    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"
Oct 29 22:58:28.276185 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:28.276128    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82" event={"ID":"0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a","Type":"ContainerDied","Data":"47a8c419c2832d4047cbf9e3bd553a3503fb73d7ca2d17df17db366f1d743f3b"}
Oct 29 22:58:28.276676 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:28.276197    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47a8c419c2832d4047cbf9e3bd553a3503fb73d7ca2d17df17db366f1d743f3b"
Oct 29 22:58:28.401368 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:28.397925    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:29.032747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.032710    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"]
Oct 29 22:58:29.046203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.046177    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bdc36f4-w5gsd"]
Oct 29 22:58:29.601871 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.601830    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"]
Oct 29 22:58:29.602924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.602898    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a" containerName="revoke-keys"
Oct 29 22:58:29.602924 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.602921    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a" containerName="revoke-keys"
Oct 29 22:58:29.603210 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.603194    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a" containerName="revoke-keys"
Oct 29 22:58:29.607479 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.607457    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:29.625366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.625305    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"]
Oct 29 22:58:29.631852 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.631828    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:29.632021 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.631852    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:58:29.662902 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.662880    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11ac04dd-ng2ld\" (UID: \"3860308b-e709-4dc4-96ad-875f5c15c551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:29.663023 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.662913    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11ac04dd-ng2ld\" (UID: \"3860308b-e709-4dc4-96ad-875f5c15c551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:29.663110 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.663090    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4lr6\" (UniqueName: \"kubernetes.io/projected/3860308b-e709-4dc4-96ad-875f5c15c551-kube-api-access-q4lr6\") pod \"maas-api-revoke-sub-11ac04dd-ng2ld\" (UID: \"3860308b-e709-4dc4-96ad-875f5c15c551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:29.764641 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.764613    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11ac04dd-ng2ld\" (UID: \"3860308b-e709-4dc4-96ad-875f5c15c551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:29.764751 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.764656    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11ac04dd-ng2ld\" (UID: \"3860308b-e709-4dc4-96ad-875f5c15c551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:29.764804 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.764752    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4lr6\" (UniqueName: \"kubernetes.io/projected/3860308b-e709-4dc4-96ad-875f5c15c551-kube-api-access-q4lr6\") pod \"maas-api-revoke-sub-11ac04dd-ng2ld\" (UID: \"3860308b-e709-4dc4-96ad-875f5c15c551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:29.765268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.765245    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11ac04dd-ng2ld\" (UID: \"3860308b-e709-4dc4-96ad-875f5c15c551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:29.767154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.767133    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11ac04dd-ng2ld\" (UID: \"3860308b-e709-4dc4-96ad-875f5c15c551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:29.786466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.786443    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4lr6\" (UniqueName: \"kubernetes.io/projected/3860308b-e709-4dc4-96ad-875f5c15c551-kube-api-access-q4lr6\") pod \"maas-api-revoke-sub-11ac04dd-ng2ld\" (UID: \"3860308b-e709-4dc4-96ad-875f5c15c551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:29.924400 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:29.924311    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:30.137538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:30.137494    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289a6c0a-140b-471a-a549-cf3f2d07dabc" path="/var/lib/kubelet/pods/289a6c0a-140b-471a-a549-cf3f2d07dabc/volumes"
Oct 29 22:58:30.239989 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:30.239873    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"]
Oct 29 22:58:30.524167 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:30.524072    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"]
Oct 29 22:58:30.698164 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:30.698128    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:30.870724 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:30.870676    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:31.010281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:31.010244    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:31.223189 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:31.223149    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld" event={"ID":"3860308b-e709-4dc4-96ad-875f5c15c551","Type":"ContainerStarted","Data":"24ba0c485f2ac97ffa6fbaba1015717a68b6d7812ea078e2c7065a7e6e8cb650"}
Oct 29 22:58:31.223438 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:31.223223    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld" event={"ID":"3860308b-e709-4dc4-96ad-875f5c15c551","Type":"ContainerStarted","Data":"6a3cc70dc22fa3420430c4cc81f17aaf6b34391c7c0831fc17fdd44fdfbc83f7"}
Oct 29 22:58:31.290617 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:31.290543    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld" podStartSLOduration=2.290521375 podStartE2EDuration="2.290521375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:31.272767937 +0000 UTC m=+2653.297544084" watchObservedRunningTime="2026-10-29 22:58:31.290521375 +0000 UTC m=+2653.315297508"
Oct 29 22:58:31.702180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:31.702124    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:31.850645 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:31.850539    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:32.075677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:32.075649    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 22:58:32.235178 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:32.235144    4860 generic.go:356] "Generic (PLEG): container finished" podID="3860308b-e709-4dc4-96ad-875f5c15c551" containerID="24ba0c485f2ac97ffa6fbaba1015717a68b6d7812ea078e2c7065a7e6e8cb650" exitCode=0
Oct 29 22:58:32.235393 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:32.235195    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld" event={"ID":"3860308b-e709-4dc4-96ad-875f5c15c551","Type":"ContainerDied","Data":"24ba0c485f2ac97ffa6fbaba1015717a68b6d7812ea078e2c7065a7e6e8cb650"}
Oct 29 22:58:32.807282 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:32.806429    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:33.423447 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:33.423425    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:33.535590 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:33.535505    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-xks-ca\") pod \"3860308b-e709-4dc4-96ad-875f5c15c551\" (UID: \"3860308b-e709-4dc4-96ad-875f5c15c551\") "
Oct 29 22:58:33.535766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:33.535747    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-openshift-ca\") pod \"3860308b-e709-4dc4-96ad-875f5c15c551\" (UID: \"3860308b-e709-4dc4-96ad-875f5c15c551\") "
Oct 29 22:58:33.535976 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:33.535951    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3860308b-e709-4dc4-96ad-875f5c15c551-kube-api-access-q4lr6\" (UniqueName: \"kubernetes.io/projected/3860308b-e709-4dc4-96ad-875f5c15c551-kube-api-access-q4lr6\") pod \"3860308b-e709-4dc4-96ad-875f5c15c551\" (UID: \"3860308b-e709-4dc4-96ad-875f5c15c551\") "
Oct 29 22:58:33.536137 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:33.536111    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-openshift-ca" pod "3860308b-e709-4dc4-96ad-875f5c15c551" (UID: "3860308b-e709-4dc4-96ad-875f5c15c551"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:33.536856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:33.536832    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:33.537721 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:33.537693    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-xks-ca" pod "3860308b-e709-4dc4-96ad-875f5c15c551" (UID: "3860308b-e709-4dc4-96ad-875f5c15c551"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:33.538313 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:33.538287    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3860308b-e709-4dc4-96ad-875f5c15c551-kube-api-access-q4lr6" pod "3860308b-e709-4dc4-96ad-875f5c15c551" (UID: "3860308b-e709-4dc4-96ad-875f5c15c551"). InnerVolumeSpecName "kube-api-access-q4lr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:33.649367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:33.639657    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4lr6\" (UniqueName: \"kubernetes.io/projected/3860308b-e709-4dc4-96ad-875f5c15c551-kube-api-access-q4lr6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:33.649367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:33.639687    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3860308b-e709-4dc4-96ad-875f5c15c551-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:34.255893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:34.255863    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"
Oct 29 22:58:34.256278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:34.255894    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld" event={"ID":"3860308b-e709-4dc4-96ad-875f5c15c551","Type":"ContainerDied","Data":"6a3cc70dc22fa3420430c4cc81f17aaf6b34391c7c0831fc17fdd44fdfbc83f7"}
Oct 29 22:58:34.256278 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:34.255941    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a3cc70dc22fa3420430c4cc81f17aaf6b34391c7c0831fc17fdd44fdfbc83f7"
Oct 29 22:58:34.318806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:34.318782    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:35.525423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:35.525367    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz" podUID="ce01c801-a808-4a2b-bc03-2538b117742e" containerName="payload-pre-processing" containerID="cri-o://3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239" gracePeriod=25
Oct 29 22:58:35.830540 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:35.830513    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"
Oct 29 22:58:35.952769 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:35.952735    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:35.976427 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:35.976380    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce01c801-a808-4a2b-bc03-2538b117742e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce01c801-a808-4a2b-bc03-2538b117742e-plugins-config-volume\") pod \"ce01c801-a808-4a2b-bc03-2538b117742e\" (UID: \"ce01c801-a808-4a2b-bc03-2538b117742e\") "
Oct 29 22:58:35.976609 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:35.976529    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce01c801-a808-4a2b-bc03-2538b117742e-kube-api-access-fg52v\" (UniqueName: \"kubernetes.io/projected/ce01c801-a808-4a2b-bc03-2538b117742e-kube-api-access-fg52v\") pod \"ce01c801-a808-4a2b-bc03-2538b117742e\" (UID: \"ce01c801-a808-4a2b-bc03-2538b117742e\") "
Oct 29 22:58:35.978211 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:35.978162    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce01c801-a808-4a2b-bc03-2538b117742e-plugins-config-volume" pod "ce01c801-a808-4a2b-bc03-2538b117742e" (UID: "ce01c801-a808-4a2b-bc03-2538b117742e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:35.980339 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:35.980293    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce01c801-a808-4a2b-bc03-2538b117742e-kube-api-access-fg52v" pod "ce01c801-a808-4a2b-bc03-2538b117742e" (UID: "ce01c801-a808-4a2b-bc03-2538b117742e"). InnerVolumeSpecName "kube-api-access-fg52v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:36.079301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:36.079263    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg52v\" (UniqueName: \"kubernetes.io/projected/ce01c801-a808-4a2b-bc03-2538b117742e-kube-api-access-fg52v\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:36.079301 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:36.079294    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce01c801-a808-4a2b-bc03-2538b117742e-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:36.276229 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:36.276194    4860 generic.go:356] "Generic (PLEG): container finished" podID="ce01c801-a808-4a2b-bc03-2538b117742e" containerID="3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239" exitCode=0
Oct 29 22:58:36.276448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:36.276245    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"
Oct 29 22:58:36.276448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:36.276267    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz" event={"ID":"ce01c801-a808-4a2b-bc03-2538b117742e","Type":"ContainerDied","Data":"3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239"}
Oct 29 22:58:36.276448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:36.276341    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz" event={"ID":"ce01c801-a808-4a2b-bc03-2538b117742e","Type":"ContainerDied","Data":"124ad9b44c716946937b1583612d1019d80a2d703965b5767c32bd53b615ce86"}
Oct 29 22:58:36.276448 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:36.276373    4860 scope.go:122] "RemoveContainer" containerID="3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239"
Oct 29 22:58:36.288950 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:36.288934    4860 scope.go:122] "RemoveContainer" containerID="3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239"
Oct 29 22:58:36.289199 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:58:36.289180    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239\": container with ID starting with 3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239 not found: ID does not exist" containerID="3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239"
Oct 29 22:58:36.289261 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:36.289206    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239"} err="failed to get container status \"3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239\": rpc error: code = NotFound desc = could not find container \"3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239\": container with ID starting with 3a0fdf7b44cdcd4eefff40829e7fdf21867a3cfd3643e72240b02a6ae61c3239 not found: ID does not exist"
Oct 29 22:58:36.340214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:36.340134    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"]
Oct 29 22:58:36.354493 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:36.354466    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b8f54c-59d54cdb67-t4sjz"]
Oct 29 22:58:37.507825 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:37.507789    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:38.117317 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:38.117279    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce01c801-a808-4a2b-bc03-2538b117742e" path="/var/lib/kubelet/pods/ce01c801-a808-4a2b-bc03-2538b117742e/volumes"
Oct 29 22:58:39.199461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:39.199427    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:39.963376 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:39.963341    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"]
Oct 29 22:58:39.964183 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:39.964167    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ce01c801-a808-4a2b-bc03-2538b117742e" containerName="payload-pre-processing"
Oct 29 22:58:39.964230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:39.964184    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce01c801-a808-4a2b-bc03-2538b117742e" containerName="payload-pre-processing"
Oct 29 22:58:39.964230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:39.964205    4860 cpu_manager.go:414] "containerMap: removing container" podUID="3860308b-e709-4dc4-96ad-875f5c15c551" containerName="revoke-keys"
Oct 29 22:58:39.964230 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:39.964211    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3860308b-e709-4dc4-96ad-875f5c15c551" containerName="revoke-keys"
Oct 29 22:58:39.964414 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:39.964404    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce01c801-a808-4a2b-bc03-2538b117742e" containerName="payload-pre-processing"
Oct 29 22:58:39.964459 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:39.964416    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="3860308b-e709-4dc4-96ad-875f5c15c551" containerName="revoke-keys"
Oct 29 22:58:39.969934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:39.969903    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:39.986909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:39.986881    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"]
Oct 29 22:58:39.992833 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:39.992806    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:39.992955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:39.992877    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:58:40.039104 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:40.039067    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv8nq\" (UniqueName: \"kubernetes.io/projected/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-kube-api-access-bv8nq\") pod \"maas-api-revoke-sub-e7dc7eff-7rnvs\" (UID: \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:40.039291 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:40.039194    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7dc7eff-7rnvs\" (UID: \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:40.039475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:40.039372    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7dc7eff-7rnvs\" (UID: \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:40.140461 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:40.140430    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv8nq\" (UniqueName: \"kubernetes.io/projected/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-kube-api-access-bv8nq\") pod \"maas-api-revoke-sub-e7dc7eff-7rnvs\" (UID: \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:40.140661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:40.140490    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7dc7eff-7rnvs\" (UID: \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:40.140661 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:40.140535    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7dc7eff-7rnvs\" (UID: \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:40.141232 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:40.141207    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7dc7eff-7rnvs\" (UID: \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:40.143131 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:40.143105    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7dc7eff-7rnvs\" (UID: \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:40.161533 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:40.161502    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv8nq\" (UniqueName: \"kubernetes.io/projected/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-kube-api-access-bv8nq\") pod \"maas-api-revoke-sub-e7dc7eff-7rnvs\" (UID: \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:40.290106 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:40.290028    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:40.600249 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:58:40.600215    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96fea28_eee8_4c7d_8c3b_435d81aa80f6.slice/crio-7236fcbb8e44a48a099b43371fd3b8705ab2034882d4f3632781516a56e3a246 WatchSource:0}: Error finding container 7236fcbb8e44a48a099b43371fd3b8705ab2034882d4f3632781516a56e3a246: Status 404 returned error can't find the container with id 7236fcbb8e44a48a099b43371fd3b8705ab2034882d4f3632781516a56e3a246
Oct 29 22:58:40.604245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:40.604219    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"]
Oct 29 22:58:40.716204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:40.716176    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:41.450868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:41.446783    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs" event={"ID":"f96fea28-eee8-4c7d-8c3b-435d81aa80f6","Type":"ContainerStarted","Data":"a6c2c9e7750b2061f47e413ae864063567ae4693fb1a8431cce01cef1666b8fe"}
Oct 29 22:58:41.450868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:41.446866    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs" event={"ID":"f96fea28-eee8-4c7d-8c3b-435d81aa80f6","Type":"ContainerStarted","Data":"7236fcbb8e44a48a099b43371fd3b8705ab2034882d4f3632781516a56e3a246"}
Oct 29 22:58:41.562220 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:41.561268    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs" podStartSLOduration=2.561247249 podStartE2EDuration="2.561247249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:41.550718819 +0000 UTC m=+2663.575494962" watchObservedRunningTime="2026-10-29 22:58:41.561247249 +0000 UTC m=+2663.586023382"
Oct 29 22:58:42.463174 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:42.463137    4860 generic.go:356] "Generic (PLEG): container finished" podID="f96fea28-eee8-4c7d-8c3b-435d81aa80f6" containerID="a6c2c9e7750b2061f47e413ae864063567ae4693fb1a8431cce01cef1666b8fe" exitCode=0
Oct 29 22:58:42.463709 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:42.463223    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs" event={"ID":"f96fea28-eee8-4c7d-8c3b-435d81aa80f6","Type":"ContainerDied","Data":"a6c2c9e7750b2061f47e413ae864063567ae4693fb1a8431cce01cef1666b8fe"}
Oct 29 22:58:42.535124 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:42.535087    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:44.215909 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.215878    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:44.367687 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.367609    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-kube-api-access-bv8nq\" (UniqueName: \"kubernetes.io/projected/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-kube-api-access-bv8nq\") pod \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\" (UID: \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\") "
Oct 29 22:58:44.367859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.367745    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-xks-ca\") pod \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\" (UID: \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\") "
Oct 29 22:58:44.367859 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.367816    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-openshift-ca\") pod \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\" (UID: \"f96fea28-eee8-4c7d-8c3b-435d81aa80f6\") "
Oct 29 22:58:44.368245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.368212    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-openshift-ca" pod "f96fea28-eee8-4c7d-8c3b-435d81aa80f6" (UID: "f96fea28-eee8-4c7d-8c3b-435d81aa80f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:44.368636 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.368614    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:44.369885 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.369854    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-xks-ca" pod "f96fea28-eee8-4c7d-8c3b-435d81aa80f6" (UID: "f96fea28-eee8-4c7d-8c3b-435d81aa80f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:44.369987 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.369946    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-kube-api-access-bv8nq" pod "f96fea28-eee8-4c7d-8c3b-435d81aa80f6" (UID: "f96fea28-eee8-4c7d-8c3b-435d81aa80f6"). InnerVolumeSpecName "kube-api-access-bv8nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:44.470699 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.470668    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:44.470807 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.470704    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv8nq\" (UniqueName: \"kubernetes.io/projected/f96fea28-eee8-4c7d-8c3b-435d81aa80f6-kube-api-access-bv8nq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:58:44.582732 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.582703    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:44.673870 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.673790    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"
Oct 29 22:58:44.674034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.673795    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs" event={"ID":"f96fea28-eee8-4c7d-8c3b-435d81aa80f6","Type":"ContainerDied","Data":"7236fcbb8e44a48a099b43371fd3b8705ab2034882d4f3632781516a56e3a246"}
Oct 29 22:58:44.674034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:44.673935    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7236fcbb8e44a48a099b43371fd3b8705ab2034882d4f3632781516a56e3a246"
Oct 29 22:58:46.183893 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:46.183842    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:46.675355 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:46.675256    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:47.300704 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:47.300670    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:47.793724 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:47.793685    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:48.053423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:48.051541    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:48.214147 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:48.214101    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:48.241983 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:58:48.241952    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:58:49.303290 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:49.303240    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:49.432933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:49.432897    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:49.902790 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:49.902752    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:51.655352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:51.655282    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:52.439338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:52.439290    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:52.585892 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:52.585852    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:52.995267 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:52.995233    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:53.194349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.162572    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:53.562500 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.562461    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"]
Oct 29 22:58:53.581963 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.581922    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f96fea28-eee8-4c7d-8c3b-435d81aa80f6" containerName="revoke-keys"
Oct 29 22:58:53.582200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.582181    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96fea28-eee8-4c7d-8c3b-435d81aa80f6" containerName="revoke-keys"
Oct 29 22:58:53.582618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.582598    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96fea28-eee8-4c7d-8c3b-435d81aa80f6" containerName="revoke-keys"
Oct 29 22:58:53.598868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.598679    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"]
Oct 29 22:58:53.599272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.599253    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"
Oct 29 22:58:53.633800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.633226    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-85862e-dockercfg-8l9jz\""
Oct 29 22:58:53.633800 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.633344    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-85862e\""
Oct 29 22:58:53.743681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.743652    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl2r5\" (UniqueName: \"kubernetes.io/projected/b2e82b87-9570-42a9-b317-27c096817546-kube-api-access-sl2r5\") pod \"payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht\" (UID: \"b2e82b87-9570-42a9-b317-27c096817546\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"
Oct 29 22:58:53.743982 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.743958    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e82b87-9570-42a9-b317-27c096817546-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht\" (UID: \"b2e82b87-9570-42a9-b317-27c096817546\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"
Oct 29 22:58:53.845934 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.845852    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e82b87-9570-42a9-b317-27c096817546-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht\" (UID: \"b2e82b87-9570-42a9-b317-27c096817546\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"
Oct 29 22:58:53.846084 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.845995    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2r5\" (UniqueName: \"kubernetes.io/projected/b2e82b87-9570-42a9-b317-27c096817546-kube-api-access-sl2r5\") pod \"payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht\" (UID: \"b2e82b87-9570-42a9-b317-27c096817546\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"
Oct 29 22:58:53.846728 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.846698    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e82b87-9570-42a9-b317-27c096817546-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht\" (UID: \"b2e82b87-9570-42a9-b317-27c096817546\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"
Oct 29 22:58:53.868598 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.868575    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2r5\" (UniqueName: \"kubernetes.io/projected/b2e82b87-9570-42a9-b317-27c096817546-kube-api-access-sl2r5\") pod \"payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht\" (UID: \"b2e82b87-9570-42a9-b317-27c096817546\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"
Oct 29 22:58:53.941946 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:53.941925    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"
Oct 29 22:58:54.246295 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:58:54.246266    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e82b87_9570_42a9_b317_27c096817546.slice/crio-d42ceaa9ca7864050484bd5794a56879c48cc6062eebc1be93479faa41200533 WatchSource:0}: Error finding container d42ceaa9ca7864050484bd5794a56879c48cc6062eebc1be93479faa41200533: Status 404 returned error can't find the container with id d42ceaa9ca7864050484bd5794a56879c48cc6062eebc1be93479faa41200533
Oct 29 22:58:54.250972 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:54.250945    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"]
Oct 29 22:58:54.432462 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:54.432417    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:54.776730 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:54.776684    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht" event={"ID":"b2e82b87-9570-42a9-b317-27c096817546","Type":"ContainerStarted","Data":"f64436e842025356ccfd9aff65525918eb8eaa203c5473e2dded2801816171ac"}
Oct 29 22:58:54.776935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:54.776776    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht" event={"ID":"b2e82b87-9570-42a9-b317-27c096817546","Type":"ContainerStarted","Data":"d42ceaa9ca7864050484bd5794a56879c48cc6062eebc1be93479faa41200533"}
Oct 29 22:58:54.776935 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:54.776806    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"
Oct 29 22:58:54.824948 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:54.824901    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht" podStartSLOduration=1.824888297 podStartE2EDuration="1.824888297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:54.819069864 +0000 UTC m=+2676.843845996" watchObservedRunningTime="2026-10-29 22:58:54.824888297 +0000 UTC m=+2676.849664427"
Oct 29 22:58:54.967201 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:54.967163    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:55.203962 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:55.203931    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:55.555862 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:55.555776    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:56.549762 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:56.545394    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:58.256017 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:58.255983    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:59.797897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:58:59.797845    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:01.356022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:01.355984    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:02.729091 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:02.729060    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:03.252350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:03.213170    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:03.382736 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:03.382596    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:03.386369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:03.386343    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"]
Oct 29 22:59:03.395933 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:03.395898    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-06a95f-d7m8l"]
Oct 29 22:59:03.863734 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:03.863696    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:03.980379 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:03.980272    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:04.066466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:04.066433    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"]
Oct 29 22:59:04.080298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:04.080264    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4796a9-hfn82"]
Oct 29 22:59:04.097868 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:04.097829    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:04.118975 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:04.118903    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a" path="/var/lib/kubelet/pods/0701f3d8-5c09-4cca-b5e2-c21c3fa16c4a/volumes"
Oct 29 22:59:04.119775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:04.119749    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1972ec-5c66-4862-a66c-b623403a4414" path="/var/lib/kubelet/pods/9e1972ec-5c66-4862-a66c-b623403a4414/volumes"
Oct 29 22:59:04.553310 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:04.553273    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:05.817528 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:05.817498    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"
Oct 29 22:59:06.167776 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:06.167677    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:07.783374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:07.783317    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:09.386522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:09.386485    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:09.928907 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:09.928870    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:10.058891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:10.058856    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:10.213676 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:10.213640    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"]
Oct 29 22:59:10.213891 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:10.213865    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" podUID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" containerName="istio-proxy" containerID="cri-o://1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc" gracePeriod=30
Oct 29 22:59:10.283715 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:10.283686    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:10.768882 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:10.768847    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"]
Oct 29 22:59:10.775527 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:10.775498    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:10.792845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:10.792822    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:59:10.798012 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:10.797992    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"]
Oct 29 22:59:10.805629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:10.805606    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:59:10.901618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:10.901584    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv\" (UID: \"d31c7359-34ad-4556-b6a2-2f7edf281c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:10.901826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:10.901756    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pjcv\" (UniqueName: \"kubernetes.io/projected/d31c7359-34ad-4556-b6a2-2f7edf281c16-kube-api-access-8pjcv\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv\" (UID: \"d31c7359-34ad-4556-b6a2-2f7edf281c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:10.901826 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:10.901795    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv\" (UID: \"d31c7359-34ad-4556-b6a2-2f7edf281c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:11.003446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:11.003415    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv\" (UID: \"d31c7359-34ad-4556-b6a2-2f7edf281c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:11.003633 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:11.003612    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pjcv\" (UniqueName: \"kubernetes.io/projected/d31c7359-34ad-4556-b6a2-2f7edf281c16-kube-api-access-8pjcv\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv\" (UID: \"d31c7359-34ad-4556-b6a2-2f7edf281c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:11.003682 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:11.003663    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv\" (UID: \"d31c7359-34ad-4556-b6a2-2f7edf281c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:11.004380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:11.004353    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv\" (UID: \"d31c7359-34ad-4556-b6a2-2f7edf281c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:11.005905 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:11.005884    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv\" (UID: \"d31c7359-34ad-4556-b6a2-2f7edf281c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:11.025788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:11.025761    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pjcv\" (UniqueName: \"kubernetes.io/projected/d31c7359-34ad-4556-b6a2-2f7edf281c16-kube-api-access-8pjcv\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv\" (UID: \"d31c7359-34ad-4556-b6a2-2f7edf281c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:11.090608 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:11.090581    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:11.426829 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:59:11.426796    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31c7359_34ad_4556_b6a2_2f7edf281c16.slice/crio-4f6ec5b574e905a5f306b3b153d1f50b7fcdb2d5850d00cdf34e8420b252c431 WatchSource:0}: Error finding container 4f6ec5b574e905a5f306b3b153d1f50b7fcdb2d5850d00cdf34e8420b252c431: Status 404 returned error can't find the container with id 4f6ec5b574e905a5f306b3b153d1f50b7fcdb2d5850d00cdf34e8420b252c431
Oct 29 22:59:11.427048 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:11.426915    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"]
Oct 29 22:59:11.441762 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:11.441723    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:12.047377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:12.047268    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv" event={"ID":"d31c7359-34ad-4556-b6a2-2f7edf281c16","Type":"ContainerStarted","Data":"e8eefd8ecc9da1186c30762eeb3645a3a6ae2b4f19557d5467efec8478142c8d"}
Oct 29 22:59:12.047377 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:12.047362    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv" event={"ID":"d31c7359-34ad-4556-b6a2-2f7edf281c16","Type":"ContainerStarted","Data":"4f6ec5b574e905a5f306b3b153d1f50b7fcdb2d5850d00cdf34e8420b252c431"}
Oct 29 22:59:12.099738 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:12.099685    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv" podStartSLOduration=2.099669292 podStartE2EDuration="2.099669292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:12.092821767 +0000 UTC m=+2694.117597923" watchObservedRunningTime="2026-10-29 22:59:12.099669292 +0000 UTC m=+2694.124445422"
Oct 29 22:59:12.492090 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:12.491908    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:12.690432 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:12.690396    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:12.887403 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:12.882951    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:13.174407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:13.172353    4860 generic.go:356] "Generic (PLEG): container finished" podID="d31c7359-34ad-4556-b6a2-2f7edf281c16" containerID="e8eefd8ecc9da1186c30762eeb3645a3a6ae2b4f19557d5467efec8478142c8d" exitCode=0
Oct 29 22:59:13.174407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:13.172496    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv" event={"ID":"d31c7359-34ad-4556-b6a2-2f7edf281c16","Type":"ContainerDied","Data":"e8eefd8ecc9da1186c30762eeb3645a3a6ae2b4f19557d5467efec8478142c8d"}
Oct 29 22:59:13.481747 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:13.481705    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:14.399180 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.399153    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:14.467566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.467485    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-xks-ca\") pod \"d31c7359-34ad-4556-b6a2-2f7edf281c16\" (UID: \"d31c7359-34ad-4556-b6a2-2f7edf281c16\") "
Oct 29 22:59:14.467566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.467530    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-openshift-ca\") pod \"d31c7359-34ad-4556-b6a2-2f7edf281c16\" (UID: \"d31c7359-34ad-4556-b6a2-2f7edf281c16\") "
Oct 29 22:59:14.467566 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.467562    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31c7359-34ad-4556-b6a2-2f7edf281c16-kube-api-access-8pjcv\" (UniqueName: \"kubernetes.io/projected/d31c7359-34ad-4556-b6a2-2f7edf281c16-kube-api-access-8pjcv\") pod \"d31c7359-34ad-4556-b6a2-2f7edf281c16\" (UID: \"d31c7359-34ad-4556-b6a2-2f7edf281c16\") "
Oct 29 22:59:14.468063 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.468032    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-openshift-ca" pod "d31c7359-34ad-4556-b6a2-2f7edf281c16" (UID: "d31c7359-34ad-4556-b6a2-2f7edf281c16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:14.468594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.468566    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:14.469956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.469930    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31c7359-34ad-4556-b6a2-2f7edf281c16-kube-api-access-8pjcv" pod "d31c7359-34ad-4556-b6a2-2f7edf281c16" (UID: "d31c7359-34ad-4556-b6a2-2f7edf281c16"). InnerVolumeSpecName "kube-api-access-8pjcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:14.470894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.470867    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-xks-ca" pod "d31c7359-34ad-4556-b6a2-2f7edf281c16" (UID: "d31c7359-34ad-4556-b6a2-2f7edf281c16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:14.493357 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.493280    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:59:14.493466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.493363    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" podUID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:59:14.570066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.570038    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d31c7359-34ad-4556-b6a2-2f7edf281c16-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:14.570177 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.570072    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pjcv\" (UniqueName: \"kubernetes.io/projected/d31c7359-34ad-4556-b6a2-2f7edf281c16-kube-api-access-8pjcv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:14.698217 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.698185    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:14.705157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.705129    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"]
Oct 29 22:59:14.772281 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.772215    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"]
Oct 29 22:59:14.951504 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:14.951471    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:15.169095 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:15.168995    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:15.212369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:15.211317    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv" event={"ID":"d31c7359-34ad-4556-b6a2-2f7edf281c16","Type":"ContainerDied","Data":"4f6ec5b574e905a5f306b3b153d1f50b7fcdb2d5850d00cdf34e8420b252c431"}
Oct 29 22:59:15.212369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:15.211424    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f6ec5b574e905a5f306b3b153d1f50b7fcdb2d5850d00cdf34e8420b252c431"
Oct 29 22:59:15.212369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:15.211367    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"
Oct 29 22:59:15.324146 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:15.323684    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:15.451094 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:15.451056    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:15.887394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:15.885982    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:16.089197 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.089168    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:59:16.223044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.223011    4860 generic.go:356] "Generic (PLEG): container finished" podID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" containerID="1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc" exitCode=0
Oct 29 22:59:16.223288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.223068    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"
Oct 29 22:59:16.223288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.223094    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" event={"ID":"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346","Type":"ContainerDied","Data":"1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc"}
Oct 29 22:59:16.223288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.223155    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj" event={"ID":"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346","Type":"ContainerDied","Data":"8797dd4d727d60234b8f87acdf1ed795089256de4310bbb280b037557e3b3b21"}
Oct 29 22:59:16.223288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.223192    4860 scope.go:122] "RemoveContainer" containerID="1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc"
Oct 29 22:59:16.234511 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.234487    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-credential-socket\") pod \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") "
Oct 29 22:59:16.234635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.234552    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-token\" (UniqueName: \"kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-token\") pod \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") "
Oct 29 22:59:16.234635 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.234589    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-kube-api-access-fvqlg\" (UniqueName: \"kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-kube-api-access-fvqlg\") pod \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") "
Oct 29 22:59:16.234749 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.234725    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-envoy\") pod \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") "
Oct 29 22:59:16.234828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.234797    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-credential-socket" pod "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" (UID: "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:16.234936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.234914    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-podinfo\") pod \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") "
Oct 29 22:59:16.235025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.235009    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-certs\") pod \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") "
Oct 29 22:59:16.235128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.235112    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istiod-ca-cert\") pod \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") "
Oct 29 22:59:16.235243 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.235223    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-data\") pod \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") "
Oct 29 22:59:16.235297 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.235278    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-socket\") pod \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\" (UID: \"f5143df2-3ef6-47a5-9c4c-87d0ac7a0346\") "
Oct 29 22:59:16.235442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.235410    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-certs" pod "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" (UID: "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:16.236312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.235728    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istiod-ca-cert" pod "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" (UID: "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:16.236312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.235946    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-data" pod "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" (UID: "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:16.236312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.236167    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-socket" pod "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" (UID: "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:16.236534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.236314    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.236534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.236360    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.236534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.236373    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.236534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.236386    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.236534 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.236398    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.237588 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.237561    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-podinfo" pod "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" (UID: "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:16.237780 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.237761    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-token" pod "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" (UID: "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:16.238052 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.238027    4860 scope.go:122] "RemoveContainer" containerID="1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc"
Oct 29 22:59:16.238992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.238280    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-kube-api-access-fvqlg" pod "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" (UID: "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346"). InnerVolumeSpecName "kube-api-access-fvqlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:16.238992 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:59:16.238428    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc\": container with ID starting with 1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc not found: ID does not exist" containerID="1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc"
Oct 29 22:59:16.238992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.238454    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc"} err="failed to get container status \"1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc\": rpc error: code = NotFound desc = could not find container \"1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc\": container with ID starting with 1a9e03f48375b1125a0321673a3b4c371d9864e6addad99ae9538f2a3d5b50dc not found: ID does not exist"
Oct 29 22:59:16.238992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.238530    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-envoy" pod "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" (UID: "f5143df2-3ef6-47a5-9c4c-87d0ac7a0346"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:16.342788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.341082    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.342788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.341126    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvqlg\" (UniqueName: \"kubernetes.io/projected/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-kube-api-access-fvqlg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.342788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.341142    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.342788 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.341155    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.388960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.388927    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:16.585463 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.585432    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"]
Oct 29 22:59:16.606238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.606200    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-06a95f-openshift-default-74f57588cc-f4bjj"]
Oct 29 22:59:16.674046 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:16.674016    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:18.057334 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:59:18.057282    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:59:18.116433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:18.116399    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" path="/var/lib/kubelet/pods/f5143df2-3ef6-47a5-9c4c-87d0ac7a0346/volumes"
Oct 29 22:59:18.742955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:18.742919    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:59:18.775129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:18.775110    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 22:59:19.162247 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:19.162118    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:19.705787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:19.705747    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct" podUID="ef4421ef-787c-44cb-b6e5-e582c584582f" containerName="payload-processing" containerID="cri-o://445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13" gracePeriod=25
Oct 29 22:59:19.773535 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:19.773504    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r" podUID="fbb59757-ce69-45bb-9bea-fd9701270b86" containerName="payload-pre-processing" containerID="cri-o://319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb" gracePeriod=25
Oct 29 22:59:20.089286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.089258    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"
Oct 29 22:59:20.106298 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.106262    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4421ef-787c-44cb-b6e5-e582c584582f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4421ef-787c-44cb-b6e5-e582c584582f-plugins-config-volume\") pod \"ef4421ef-787c-44cb-b6e5-e582c584582f\" (UID: \"ef4421ef-787c-44cb-b6e5-e582c584582f\") "
Oct 29 22:59:20.106557 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.106536    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4421ef-787c-44cb-b6e5-e582c584582f-kube-api-access-bsnw6\" (UniqueName: \"kubernetes.io/projected/ef4421ef-787c-44cb-b6e5-e582c584582f-kube-api-access-bsnw6\") pod \"ef4421ef-787c-44cb-b6e5-e582c584582f\" (UID: \"ef4421ef-787c-44cb-b6e5-e582c584582f\") "
Oct 29 22:59:20.107561 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.107533    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4421ef-787c-44cb-b6e5-e582c584582f-plugins-config-volume" pod "ef4421ef-787c-44cb-b6e5-e582c584582f" (UID: "ef4421ef-787c-44cb-b6e5-e582c584582f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:20.111430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.111401    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4421ef-787c-44cb-b6e5-e582c584582f-kube-api-access-bsnw6" pod "ef4421ef-787c-44cb-b6e5-e582c584582f" (UID: "ef4421ef-787c-44cb-b6e5-e582c584582f"). InnerVolumeSpecName "kube-api-access-bsnw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:20.153475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.153446    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"
Oct 29 22:59:20.208681 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.208644    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb59757-ce69-45bb-9bea-fd9701270b86-kube-api-access-f97fl\" (UniqueName: \"kubernetes.io/projected/fbb59757-ce69-45bb-9bea-fd9701270b86-kube-api-access-f97fl\") pod \"fbb59757-ce69-45bb-9bea-fd9701270b86\" (UID: \"fbb59757-ce69-45bb-9bea-fd9701270b86\") "
Oct 29 22:59:20.209204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.209013    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb59757-ce69-45bb-9bea-fd9701270b86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb59757-ce69-45bb-9bea-fd9701270b86-plugins-config-volume\") pod \"fbb59757-ce69-45bb-9bea-fd9701270b86\" (UID: \"fbb59757-ce69-45bb-9bea-fd9701270b86\") "
Oct 29 22:59:20.209473 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.209449    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb59757-ce69-45bb-9bea-fd9701270b86-plugins-config-volume" pod "fbb59757-ce69-45bb-9bea-fd9701270b86" (UID: "fbb59757-ce69-45bb-9bea-fd9701270b86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:20.209770 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.209750    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4421ef-787c-44cb-b6e5-e582c584582f-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:20.209850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.209774    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb59757-ce69-45bb-9bea-fd9701270b86-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:20.209850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.209787    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsnw6\" (UniqueName: \"kubernetes.io/projected/ef4421ef-787c-44cb-b6e5-e582c584582f-kube-api-access-bsnw6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:20.210487 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.210462    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:20.211066 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.211046    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb59757-ce69-45bb-9bea-fd9701270b86-kube-api-access-f97fl" pod "fbb59757-ce69-45bb-9bea-fd9701270b86" (UID: "fbb59757-ce69-45bb-9bea-fd9701270b86"). InnerVolumeSpecName "kube-api-access-f97fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:20.265041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.264970    4860 generic.go:356] "Generic (PLEG): container finished" podID="ef4421ef-787c-44cb-b6e5-e582c584582f" containerID="445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13" exitCode=0
Oct 29 22:59:20.265041 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.265017    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"
Oct 29 22:59:20.265248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.265061    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct" event={"ID":"ef4421ef-787c-44cb-b6e5-e582c584582f","Type":"ContainerDied","Data":"445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13"}
Oct 29 22:59:20.265248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.265125    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct" event={"ID":"ef4421ef-787c-44cb-b6e5-e582c584582f","Type":"ContainerDied","Data":"cb403c4393bd46dee301ca5be836baaf70b2549c4236f999bc0258ae2823f80c"}
Oct 29 22:59:20.265248 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.265159    4860 scope.go:122] "RemoveContainer" containerID="445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13"
Oct 29 22:59:20.266785 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.266761    4860 generic.go:356] "Generic (PLEG): container finished" podID="fbb59757-ce69-45bb-9bea-fd9701270b86" containerID="319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb" exitCode=0
Oct 29 22:59:20.266923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.266843    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"
Oct 29 22:59:20.266923 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.266890    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r" event={"ID":"fbb59757-ce69-45bb-9bea-fd9701270b86","Type":"ContainerDied","Data":"319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb"}
Oct 29 22:59:20.267044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.266930    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r" event={"ID":"fbb59757-ce69-45bb-9bea-fd9701270b86","Type":"ContainerDied","Data":"a218f25bac28986caaffec050f07f1fbe1ffc4649693fb56f2c6b531d21b8c8a"}
Oct 29 22:59:20.282483 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.282468    4860 scope.go:122] "RemoveContainer" containerID="445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13"
Oct 29 22:59:20.282799 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:59:20.282779    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13\": container with ID starting with 445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13 not found: ID does not exist" containerID="445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13"
Oct 29 22:59:20.282917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.282804    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13"} err="failed to get container status \"445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13\": rpc error: code = NotFound desc = could not find container \"445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13\": container with ID starting with 445e4e67ec05908888fa321f340a7ebe7584c1fdf70a14355a51c0d9f2225a13 not found: ID does not exist"
Oct 29 22:59:20.282917 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.282821    4860 scope.go:122] "RemoveContainer" containerID="319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb"
Oct 29 22:59:20.296187 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.296170    4860 scope.go:122] "RemoveContainer" containerID="319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb"
Oct 29 22:59:20.296498 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:59:20.296480    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb\": container with ID starting with 319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb not found: ID does not exist" containerID="319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb"
Oct 29 22:59:20.296559 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.296505    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb"} err="failed to get container status \"319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb\": rpc error: code = NotFound desc = could not find container \"319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb\": container with ID starting with 319f15cfc9055c38565b3816d03cc2a1c4eee89381dbd2a2229b3c314f1070bb not found: ID does not exist"
Oct 29 22:59:20.311454 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.311427    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f97fl\" (UniqueName: \"kubernetes.io/projected/fbb59757-ce69-45bb-9bea-fd9701270b86-kube-api-access-f97fl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:20.340398 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.340369    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"]
Oct 29 22:59:20.348044 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.348021    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-57b979dcd7-zqf6r"]
Oct 29 22:59:20.400148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.400124    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"]
Oct 29 22:59:20.406235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.406208    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-83f120-59d8965d64-mksct"]
Oct 29 22:59:20.410206 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:20.410175    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:21.160160 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.160002    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"]
Oct 29 22:59:21.169442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.169385    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" containerName="istio-proxy"
Oct 29 22:59:21.169442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.169417    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" containerName="istio-proxy"
Oct 29 22:59:21.169442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.169432    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d31c7359-34ad-4556-b6a2-2f7edf281c16" containerName="revoke-keys"
Oct 29 22:59:21.169442 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.169441    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31c7359-34ad-4556-b6a2-2f7edf281c16" containerName="revoke-keys"
Oct 29 22:59:21.169878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.169495    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ef4421ef-787c-44cb-b6e5-e582c584582f" containerName="payload-processing"
Oct 29 22:59:21.169878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.169503    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4421ef-787c-44cb-b6e5-e582c584582f" containerName="payload-processing"
Oct 29 22:59:21.169878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.169542    4860 cpu_manager.go:414] "containerMap: removing container" podUID="fbb59757-ce69-45bb-9bea-fd9701270b86" containerName="payload-pre-processing"
Oct 29 22:59:21.169878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.169550    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb59757-ce69-45bb-9bea-fd9701270b86" containerName="payload-pre-processing"
Oct 29 22:59:21.169878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.169767    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4421ef-787c-44cb-b6e5-e582c584582f" containerName="payload-processing"
Oct 29 22:59:21.169878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.169783    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb59757-ce69-45bb-9bea-fd9701270b86" containerName="payload-pre-processing"
Oct 29 22:59:21.169878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.169802    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5143df2-3ef6-47a5-9c4c-87d0ac7a0346" containerName="istio-proxy"
Oct 29 22:59:21.169878 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.169811    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31c7359-34ad-4556-b6a2-2f7edf281c16" containerName="revoke-keys"
Oct 29 22:59:21.187474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.185501    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:21.188367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.188292    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"]
Oct 29 22:59:21.192001 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.191962    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:21.220475 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.219974    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:59:21.238594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.238563    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 22:59:21.245670 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.245632    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m\" (UID: \"9bbf7823-986b-41d2-8904-8c5e66811426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:21.245845 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.245767    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtptx\" (UniqueName: \"kubernetes.io/projected/9bbf7823-986b-41d2-8904-8c5e66811426-kube-api-access-mtptx\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m\" (UID: \"9bbf7823-986b-41d2-8904-8c5e66811426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:21.246631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.246601    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m\" (UID: \"9bbf7823-986b-41d2-8904-8c5e66811426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:21.347938 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.347901    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m\" (UID: \"9bbf7823-986b-41d2-8904-8c5e66811426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:21.348117 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.347975    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtptx\" (UniqueName: \"kubernetes.io/projected/9bbf7823-986b-41d2-8904-8c5e66811426-kube-api-access-mtptx\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m\" (UID: \"9bbf7823-986b-41d2-8904-8c5e66811426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:21.348372 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.348341    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m\" (UID: \"9bbf7823-986b-41d2-8904-8c5e66811426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:21.349006 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.348988    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m\" (UID: \"9bbf7823-986b-41d2-8904-8c5e66811426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:21.350299 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.350280    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m\" (UID: \"9bbf7823-986b-41d2-8904-8c5e66811426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:21.375165 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.375141    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtptx\" (UniqueName: \"kubernetes.io/projected/9bbf7823-986b-41d2-8904-8c5e66811426-kube-api-access-mtptx\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m\" (UID: \"9bbf7823-986b-41d2-8904-8c5e66811426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:21.510245 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.510214    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:21.760417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.760343    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"]
Oct 29 22:59:21.760564 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.760529    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj" podUID="4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" containerName="istio-proxy" containerID="cri-o://50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19" gracePeriod=30
Oct 29 22:59:21.788717 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.788681    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:21.836430 ip-10-0-128-31 kubenswrapper[4860]: W1029 22:59:21.836395    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbf7823_986b_41d2_8904_8c5e66811426.slice/crio-89685b0bf34e6019910730668c886b94bef4d2e31c5829bfd2a79698898da1df WatchSource:0}: Error finding container 89685b0bf34e6019910730668c886b94bef4d2e31c5829bfd2a79698898da1df: Status 404 returned error can't find the container with id 89685b0bf34e6019910730668c886b94bef4d2e31c5829bfd2a79698898da1df
Oct 29 22:59:21.839737 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.839711    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"]
Oct 29 22:59:21.974366 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:21.969774    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:22.152358 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:22.152255    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4421ef-787c-44cb-b6e5-e582c584582f" path="/var/lib/kubelet/pods/ef4421ef-787c-44cb-b6e5-e582c584582f/volumes"
Oct 29 22:59:22.152979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:22.152956    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb59757-ce69-45bb-9bea-fd9701270b86" path="/var/lib/kubelet/pods/fbb59757-ce69-45bb-9bea-fd9701270b86/volumes"
Oct 29 22:59:22.220538 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:22.220510    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"]
Oct 29 22:59:22.274631 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:22.274601    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"]
Oct 29 22:59:22.296916 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:22.296884    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m" event={"ID":"9bbf7823-986b-41d2-8904-8c5e66811426","Type":"ContainerStarted","Data":"89685b0bf34e6019910730668c886b94bef4d2e31c5829bfd2a79698898da1df"}
Oct 29 22:59:22.411698 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:22.411618    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:22.558739 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:22.558698    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:23.089348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:23.088369    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"]
Oct 29 22:59:23.129346 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:23.128893    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c420727-65565"]
Oct 29 22:59:23.309761 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:23.309728    4860 generic.go:356] "Generic (PLEG): container finished" podID="9bbf7823-986b-41d2-8904-8c5e66811426" containerID="7b1e564ecdd5cde0cecf13c65ac17ce545abee7a3d31f48d6585054419b946db" exitCode=0
Oct 29 22:59:23.310207 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:23.309810    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m" event={"ID":"9bbf7823-986b-41d2-8904-8c5e66811426","Type":"ContainerDied","Data":"7b1e564ecdd5cde0cecf13c65ac17ce545abee7a3d31f48d6585054419b946db"}
Oct 29 22:59:23.815505 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:23.813178    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:23.989988 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:23.989959    4860 scope.go:122] "RemoveContainer" containerID="429d69c2c24a09b65362123aa844aa4c70974243b5a728fc049acbde18b35fe7"
Oct 29 22:59:24.002238 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.002220    4860 scope.go:122] "RemoveContainer" containerID="239e4c5afc9093163585e20440671a76e3ce6332178017beccaf65c7f8f26d09"
Oct 29 22:59:24.013287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.013260    4860 scope.go:122] "RemoveContainer" containerID="86ed9538d7852b702bce3a419959f93ec4e0a03a67f6aefd3faf029d1304a327"
Oct 29 22:59:24.024502 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.024478    4860 scope.go:122] "RemoveContainer" containerID="accdbfce51366d3fcbc23c355633d90970e8bf54d73ee1f8dbba5d6a3fe8af9c"
Oct 29 22:59:24.035311 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.035287    4860 scope.go:122] "RemoveContainer" containerID="a4e44ade94e3efc65dd161c7790e1364afe25fd4d99599f77a3b7c856bf1eda6"
Oct 29 22:59:24.045130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.045097    4860 scope.go:122] "RemoveContainer" containerID="616fab32461c552f2c8b97d578bc0994b7b05c70226699949aa9f49214872ddb"
Oct 29 22:59:24.056467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.056445    4860 scope.go:122] "RemoveContainer" containerID="f2fbc458d58b3f95c3fea0c751965b391126db409598f2ea1d0b4d74df4de41f"
Oct 29 22:59:24.118332 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.118287    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3192b217-2d8a-4c01-b959-39d74e31b70a" path="/var/lib/kubelet/pods/3192b217-2d8a-4c01-b959-39d74e31b70a/volumes"
Oct 29 22:59:24.182497 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.182473    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"]
Oct 29 22:59:24.431827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.431792    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:24.581867 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.581806    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:24.646941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.641992    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-xks-ca\") pod \"9bbf7823-986b-41d2-8904-8c5e66811426\" (UID: \"9bbf7823-986b-41d2-8904-8c5e66811426\") "
Oct 29 22:59:24.646941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.644543    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bbf7823-986b-41d2-8904-8c5e66811426-kube-api-access-mtptx\" (UniqueName: \"kubernetes.io/projected/9bbf7823-986b-41d2-8904-8c5e66811426-kube-api-access-mtptx\") pod \"9bbf7823-986b-41d2-8904-8c5e66811426\" (UID: \"9bbf7823-986b-41d2-8904-8c5e66811426\") "
Oct 29 22:59:24.646941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.644714    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-openshift-ca\") pod \"9bbf7823-986b-41d2-8904-8c5e66811426\" (UID: \"9bbf7823-986b-41d2-8904-8c5e66811426\") "
Oct 29 22:59:24.646941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.645079    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-openshift-ca" pod "9bbf7823-986b-41d2-8904-8c5e66811426" (UID: "9bbf7823-986b-41d2-8904-8c5e66811426"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:24.646941 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.646706    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:24.649786 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.649753    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-xks-ca" pod "9bbf7823-986b-41d2-8904-8c5e66811426" (UID: "9bbf7823-986b-41d2-8904-8c5e66811426"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:24.649908 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.649868    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbf7823-986b-41d2-8904-8c5e66811426-kube-api-access-mtptx" pod "9bbf7823-986b-41d2-8904-8c5e66811426" (UID: "9bbf7823-986b-41d2-8904-8c5e66811426"). InnerVolumeSpecName "kube-api-access-mtptx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:24.672629 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.672601    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:24.748268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.748243    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bbf7823-986b-41d2-8904-8c5e66811426-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:24.748268 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:24.748269    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtptx\" (UniqueName: \"kubernetes.io/projected/9bbf7823-986b-41d2-8904-8c5e66811426-kube-api-access-mtptx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:25.335132 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:25.335103    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"
Oct 29 22:59:25.335367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:25.335145    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m" event={"ID":"9bbf7823-986b-41d2-8904-8c5e66811426","Type":"ContainerDied","Data":"89685b0bf34e6019910730668c886b94bef4d2e31c5829bfd2a79698898da1df"}
Oct 29 22:59:25.335367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:25.335188    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89685b0bf34e6019910730668c886b94bef4d2e31c5829bfd2a79698898da1df"
Oct 29 22:59:25.552827 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:25.552797    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:25.978587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:25.978550    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:26.191955 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:26.191924    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:26.293697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:26.293622    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"]
Oct 29 22:59:26.306996 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:26.306965    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-db7419-szkwv"]
Oct 29 22:59:26.384130 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:26.384092    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:26.950020 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:26.949961    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:27.065129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.065098    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:59:27.136524 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.136494    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-envoy\") pod \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") "
Oct 29 22:59:27.136697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.136549    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-podinfo\") pod \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") "
Oct 29 22:59:27.136697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.136594    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-credential-socket\") pod \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") "
Oct 29 22:59:27.136697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.136636    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-certs\") pod \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") "
Oct 29 22:59:27.136697 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.136672    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-kube-api-access-jwxvl\" (UniqueName: \"kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-kube-api-access-jwxvl\") pod \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") "
Oct 29 22:59:27.136921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.136785    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-token\" (UniqueName: \"kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-token\") pod \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") "
Oct 29 22:59:27.136921 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.136891    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-credential-socket" pod "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" (UID: "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:27.137037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.136915    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-certs" pod "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" (UID: "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:27.137037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.136928    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-data\") pod \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") "
Oct 29 22:59:27.137037 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.137009    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-socket\") pod \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") "
Oct 29 22:59:27.137208 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.137127    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istiod-ca-cert\") pod \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\" (UID: \"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa\") "
Oct 29 22:59:27.137470 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.137144    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-data" pod "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" (UID: "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:27.137677 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.137551    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-socket" pod "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" (UID: "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:27.137775 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.137685    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istiod-ca-cert" pod "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" (UID: "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:27.138175 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.138150    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:27.138312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.138181    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:27.138312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.138196    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:27.138312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.138209    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:27.138312 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.138223    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:27.139100 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.139071    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-podinfo" pod "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" (UID: "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:27.139219 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.139198    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-envoy" pod "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" (UID: "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:27.139286 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.139255    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-kube-api-access-jwxvl" pod "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" (UID: "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa"). InnerVolumeSpecName "kube-api-access-jwxvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:27.139699 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.139677    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-token" pod "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" (UID: "4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:27.221407 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.221363    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2" podUID="a29d3368-acf8-4eca-a0aa-e5b11d83f05a" containerName="payload-processing" containerID="cri-o://38829dc4a397852c4629be992ef281191313ae8298cebae6695460554384594f" gracePeriod=25
Oct 29 22:59:27.239777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.239749    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:27.239884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.239778    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:27.239884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.239794    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwxvl\" (UniqueName: \"kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-kube-api-access-jwxvl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:27.239884 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.239807    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:27.277364 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.277234    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc" podUID="f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91" containerName="payload-pre-processing" containerID="cri-o://c7042a40ea8f1d51367de9b53c7638f05b0e61cf8e143f28c4381f79b13ec0e6" gracePeriod=25
Oct 29 22:59:27.433148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.432145    4860 generic.go:356] "Generic (PLEG): container finished" podID="4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" containerID="50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19" exitCode=0
Oct 29 22:59:27.433148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.432196    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj" event={"ID":"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa","Type":"ContainerDied","Data":"50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19"}
Oct 29 22:59:27.433148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.432268    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"
Oct 29 22:59:27.434079 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.433929    4860 scope.go:122] "RemoveContainer" containerID="50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19"
Oct 29 22:59:27.435436 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.435406    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj" event={"ID":"4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa","Type":"ContainerDied","Data":"4333ecd23ad76ba792ae5422011b0a750cac7bcee579bfdc77b763d4c32a0bc5"}
Oct 29 22:59:27.438435 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.438188    4860 generic.go:356] "Generic (PLEG): container finished" podID="a29d3368-acf8-4eca-a0aa-e5b11d83f05a" containerID="38829dc4a397852c4629be992ef281191313ae8298cebae6695460554384594f" exitCode=0
Oct 29 22:59:27.438435 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.438273    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2" event={"ID":"a29d3368-acf8-4eca-a0aa-e5b11d83f05a","Type":"ContainerDied","Data":"38829dc4a397852c4629be992ef281191313ae8298cebae6695460554384594f"}
Oct 29 22:59:27.440801 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.440777    4860 generic.go:356] "Generic (PLEG): container finished" podID="f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91" containerID="c7042a40ea8f1d51367de9b53c7638f05b0e61cf8e143f28c4381f79b13ec0e6" exitCode=0
Oct 29 22:59:27.440952 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.440938    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc" event={"ID":"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91","Type":"ContainerDied","Data":"c7042a40ea8f1d51367de9b53c7638f05b0e61cf8e143f28c4381f79b13ec0e6"}
Oct 29 22:59:27.452394 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.452336    4860 scope.go:122] "RemoveContainer" containerID="50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19"
Oct 29 22:59:27.452635 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:59:27.452613    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19\": container with ID starting with 50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19 not found: ID does not exist" containerID="50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19"
Oct 29 22:59:27.452716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.452640    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19"} err="failed to get container status \"50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19\": rpc error: code = NotFound desc = could not find container \"50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19\": container with ID starting with 50cf361538364cb80a864dae7d1842b7ddf96d4828d56a45e7e5b3c7330bbc19 not found: ID does not exist"
Oct 29 22:59:27.505405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.503034    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"]
Oct 29 22:59:27.515900 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.515870    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-6fc6bd-openshift-default-5665fddf89-2dhcj"]
Oct 29 22:59:27.574850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.574815    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"
Oct 29 22:59:27.630978 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.630952    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"
Oct 29 22:59:27.646830 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.646807    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-kube-api-access-4nv9b\" (UniqueName: \"kubernetes.io/projected/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-kube-api-access-4nv9b\") pod \"a29d3368-acf8-4eca-a0aa-e5b11d83f05a\" (UID: \"a29d3368-acf8-4eca-a0aa-e5b11d83f05a\") "
Oct 29 22:59:27.647148 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.647123    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-plugins-config-volume\") pod \"a29d3368-acf8-4eca-a0aa-e5b11d83f05a\" (UID: \"a29d3368-acf8-4eca-a0aa-e5b11d83f05a\") "
Oct 29 22:59:27.647619 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.647583    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-plugins-config-volume" pod "a29d3368-acf8-4eca-a0aa-e5b11d83f05a" (UID: "a29d3368-acf8-4eca-a0aa-e5b11d83f05a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:27.648034 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.648011    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:27.649305 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.649084    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-kube-api-access-4nv9b" pod "a29d3368-acf8-4eca-a0aa-e5b11d83f05a" (UID: "a29d3368-acf8-4eca-a0aa-e5b11d83f05a"). InnerVolumeSpecName "kube-api-access-4nv9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:27.748595 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.748562    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-kube-api-access-cqvh4\" (UniqueName: \"kubernetes.io/projected/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-kube-api-access-cqvh4\") pod \"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91\" (UID: \"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91\") "
Oct 29 22:59:27.748846 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.748833    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-plugins-config-volume\") pod \"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91\" (UID: \"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91\") "
Oct 29 22:59:27.749151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.749123    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-plugins-config-volume" pod "f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91" (UID: "f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:27.749404 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.749385    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:27.749486 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.749409    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nv9b\" (UniqueName: \"kubernetes.io/projected/a29d3368-acf8-4eca-a0aa-e5b11d83f05a-kube-api-access-4nv9b\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:27.750820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.750798    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-kube-api-access-cqvh4" pod "f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91" (UID: "f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91"). InnerVolumeSpecName "kube-api-access-cqvh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:27.850969 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:27.850941    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqvh4\" (UniqueName: \"kubernetes.io/projected/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91-kube-api-access-cqvh4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:28.117370 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.117264    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" path="/var/lib/kubelet/pods/4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa/volumes"
Oct 29 22:59:28.118031 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.118010    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31c7359-34ad-4556-b6a2-2f7edf281c16" path="/var/lib/kubelet/pods/d31c7359-34ad-4556-b6a2-2f7edf281c16/volumes"
Oct 29 22:59:28.454513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.454484    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"
Oct 29 22:59:28.454713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.454480    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2" event={"ID":"a29d3368-acf8-4eca-a0aa-e5b11d83f05a","Type":"ContainerDied","Data":"caad27b8c7ce0b792b8ec32aefa86c652a7c7b8b249ee321ba996a22294c7508"}
Oct 29 22:59:28.454713 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.454645    4860 scope.go:122] "RemoveContainer" containerID="38829dc4a397852c4629be992ef281191313ae8298cebae6695460554384594f"
Oct 29 22:59:28.456266 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.456216    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc" event={"ID":"f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91","Type":"ContainerDied","Data":"95b4a38cd539265551ba8dca1eaa08f576c70d8e6670395e2249ece8f71b1010"}
Oct 29 22:59:28.456431 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.456407    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"
Oct 29 22:59:28.468235 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.468219    4860 scope.go:122] "RemoveContainer" containerID="c7042a40ea8f1d51367de9b53c7638f05b0e61cf8e143f28c4381f79b13ec0e6"
Oct 29 22:59:28.526039 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.526014    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:28.527027 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.527001    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"]
Oct 29 22:59:28.534836 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.534815    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-db7419-79447f7f59-h9nmc"]
Oct 29 22:59:28.574513 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.574489    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"]
Oct 29 22:59:28.579391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:28.579371    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-db7419-84c96f7884-pqpn2"]
Oct 29 22:59:29.182805 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:29.182750    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7" podUID="90c5bd3e-fea2-497c-9e35-2d0c5a0077c9" containerName="payload-processing" containerID="cri-o://726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801" gracePeriod=25
Oct 29 22:59:30.116391 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:30.116320    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29d3368-acf8-4eca-a0aa-e5b11d83f05a" path="/var/lib/kubelet/pods/a29d3368-acf8-4eca-a0aa-e5b11d83f05a/volumes"
Oct 29 22:59:30.116919 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:30.116892    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91" path="/var/lib/kubelet/pods/f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91/volumes"
Oct 29 22:59:30.389367 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:30.389261    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:31.534865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:31.534826    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"]
Oct 29 22:59:31.571003 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:31.570973    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"]
Oct 29 22:59:31.704616 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:31.704579    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:31.845199 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:31.845088    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:31.989287 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:31.989243    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:33.036653 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.036620    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"]
Oct 29 22:59:33.050362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.050320    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2491765-zl4j8"]
Oct 29 22:59:33.550577 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.550544    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:33.779320 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.779289    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:33.931901 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.931876    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"]
Oct 29 22:59:33.932662 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.932647    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" containerName="istio-proxy"
Oct 29 22:59:33.932718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.932663    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" containerName="istio-proxy"
Oct 29 22:59:33.932718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.932682    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9bbf7823-986b-41d2-8904-8c5e66811426" containerName="revoke-keys"
Oct 29 22:59:33.932718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.932687    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bbf7823-986b-41d2-8904-8c5e66811426" containerName="revoke-keys"
Oct 29 22:59:33.932718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.932695    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a29d3368-acf8-4eca-a0aa-e5b11d83f05a" containerName="payload-processing"
Oct 29 22:59:33.932718 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.932701    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29d3368-acf8-4eca-a0aa-e5b11d83f05a" containerName="payload-processing"
Oct 29 22:59:33.932886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.932732    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91" containerName="payload-pre-processing"
Oct 29 22:59:33.932886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.932737    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91" containerName="payload-pre-processing"
Oct 29 22:59:33.932886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.932875    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72b6fe8-c4f8-4ab3-bfa6-ef4edbf77c91" containerName="payload-pre-processing"
Oct 29 22:59:33.932979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.932888    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbb58b2-9058-48c8-8dc1-0e0a2d546ffa" containerName="istio-proxy"
Oct 29 22:59:33.932979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.932896    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bbf7823-986b-41d2-8904-8c5e66811426" containerName="revoke-keys"
Oct 29 22:59:33.932979 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.932902    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29d3368-acf8-4eca-a0aa-e5b11d83f05a" containerName="payload-processing"
Oct 29 22:59:33.938433 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.938416    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:33.963209 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.963190    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-63e89c-openshift-default-dockercfg-n8dks\""
Oct 29 22:59:33.995877 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:33.995853    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"]
Oct 29 22:59:34.004362 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.004296    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7" podUID="90c5bd3e-fea2-497c-9e35-2d0c5a0077c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:34.074103 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.074068    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-socket\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.074763 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.074737    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-data\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.074960 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.074943    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-podinfo\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.075134 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.075115    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d139364-f919-4189-b3fc-cf72ade3e6d6-istiod-ca-cert\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.075294 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.075280    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-envoy\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.075522 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.075503    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-credential-socket\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.075773 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.075754    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzh9\" (UniqueName: \"kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-kube-api-access-nzzh9\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.075894 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.075881    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-certs\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.076565 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.076541    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-token\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.166347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.162897    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:34.206430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.201768    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-socket\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.206430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.201863    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-data\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.206430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.201908    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-podinfo\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.215351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.211526    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d139364-f919-4189-b3fc-cf72ade3e6d6-istiod-ca-cert\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.215351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.211598    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-envoy\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.215351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.211636    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-credential-socket\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.215351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.211773    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzh9\" (UniqueName: \"kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-kube-api-access-nzzh9\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.215351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.211809    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-certs\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.215351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.211860    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-token\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.228352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.221791    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-socket\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.228352 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.222073    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-data\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.235350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.228660    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-credential-socket\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.235350 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.229382    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d139364-f919-4189-b3fc-cf72ade3e6d6-istiod-ca-cert\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.243375 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.236745    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-certs\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.282777 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.282260    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzh9\" (UniqueName: \"kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-kube-api-access-nzzh9\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.295353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.291236    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf" path="/var/lib/kubelet/pods/2333cdd4-c0fe-4ba2-822e-3d55c11ca1cf/volumes"
Oct 29 22:59:34.295353 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.292607    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-token\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.327347 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.322235    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-envoy\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.351417 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.349545    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-podinfo\") pod \"e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") " pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.371349 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.362838    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:34.543195 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.543111    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:34.552806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.552778    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:34.750435 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.750399    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"]
Oct 29 22:59:34.750897 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.750695    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy" containerID="cri-o://56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3" gracePeriod=30
Oct 29 22:59:34.914405 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:34.895821    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:35.087782 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:35.087144    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"]
Oct 29 22:59:35.092745 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:35.092702    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:59:35.092968 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:35.092941    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:59:35.093214 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:35.093188    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 22:59:35.226186 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:35.226128    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:35.545402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:35.545293    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" event={"ID":"8d139364-f919-4189-b3fc-cf72ade3e6d6","Type":"ContainerStarted","Data":"fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6"}
Oct 29 22:59:35.545402 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:35.545396    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" event={"ID":"8d139364-f919-4189-b3fc-cf72ade3e6d6","Type":"ContainerStarted","Data":"dcfe72e95ab809adc3f577060a14da13dc33f20242cdc98b1245c789bd61ceb7"}
Oct 29 22:59:35.553568 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:35.553546    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:35.607658 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:35.607610    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" podStartSLOduration=2.60759661 podStartE2EDuration="2.60759661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:35.602650491 +0000 UTC m=+2717.627426642" watchObservedRunningTime="2026-10-29 22:59:35.60759661 +0000 UTC m=+2717.632372741"
Oct 29 22:59:35.996151 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:35.996114    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:36.536446 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:36.536397    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m" podUID="d55a1f4c-beeb-4c8d-bccb-d07d86dda333" containerName="payload-processing" containerID="cri-o://99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4" gracePeriod=25
Oct 29 22:59:36.572591 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:36.572548    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72" podUID="2c8afc25-6b5d-4136-8b69-0ef94a9ae866" containerName="payload-pre-processing" containerID="cri-o://94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086" gracePeriod=25
Oct 29 22:59:36.663865 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:36.646787    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:36.706224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:36.706195    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:36.708386 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:36.708269    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 22:59:37.920695 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:37.920655    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:38.264939 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:38.264901    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:38.524129 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:38.524038    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:39.129348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:39.128482    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:39.129348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:39.129268    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:59:39.173822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:39.133031    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:39.173822 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:39.133932    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:39.189530 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:39.184718    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Oct 29 22:59:39.330120 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:39.330077    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:39.948536 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:39.948482    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:40.081760 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.081707    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:40.088841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.088818    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:59:40.143599 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.143563    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-envoy\") pod \"a551643a-4108-4d0d-be08-6add34edffb1\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") "
Oct 29 22:59:40.144096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.143667    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a551643a-4108-4d0d-be08-6add34edffb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a551643a-4108-4d0d-be08-6add34edffb1-istiod-ca-cert\") pod \"a551643a-4108-4d0d-be08-6add34edffb1\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") "
Oct 29 22:59:40.144096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.143839    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-socket\") pod \"a551643a-4108-4d0d-be08-6add34edffb1\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") "
Oct 29 22:59:40.144096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.143881    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-kube-api-access-6gptr\" (UniqueName: \"kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-kube-api-access-6gptr\") pod \"a551643a-4108-4d0d-be08-6add34edffb1\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") "
Oct 29 22:59:40.144096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.143929    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-istio-token\" (UniqueName: \"kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-istio-token\") pod \"a551643a-4108-4d0d-be08-6add34edffb1\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") "
Oct 29 22:59:40.144096 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.144077    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-credential-socket\") pod \"a551643a-4108-4d0d-be08-6add34edffb1\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") "
Oct 29 22:59:40.144430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.144206    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-data\") pod \"a551643a-4108-4d0d-be08-6add34edffb1\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") "
Oct 29 22:59:40.144430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.144259    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a551643a-4108-4d0d-be08-6add34edffb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a551643a-4108-4d0d-be08-6add34edffb1-istio-podinfo\") pod \"a551643a-4108-4d0d-be08-6add34edffb1\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") "
Oct 29 22:59:40.144430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.144207    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a551643a-4108-4d0d-be08-6add34edffb1-istiod-ca-cert" pod "a551643a-4108-4d0d-be08-6add34edffb1" (UID: "a551643a-4108-4d0d-be08-6add34edffb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:40.144430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.144272    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-socket" pod "a551643a-4108-4d0d-be08-6add34edffb1" (UID: "a551643a-4108-4d0d-be08-6add34edffb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:40.144430 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.144303    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-certs\") pod \"a551643a-4108-4d0d-be08-6add34edffb1\" (UID: \"a551643a-4108-4d0d-be08-6add34edffb1\") "
Oct 29 22:59:40.145820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.144998    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-credential-socket" pod "a551643a-4108-4d0d-be08-6add34edffb1" (UID: "a551643a-4108-4d0d-be08-6add34edffb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:40.145820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.145468    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-certs" pod "a551643a-4108-4d0d-be08-6add34edffb1" (UID: "a551643a-4108-4d0d-be08-6add34edffb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:40.145820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.145751    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a551643a-4108-4d0d-be08-6add34edffb1-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.145820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.145774    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.145820 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.145787    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.146418 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.146390    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-data" pod "a551643a-4108-4d0d-be08-6add34edffb1" (UID: "a551643a-4108-4d0d-be08-6add34edffb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:40.147081 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.147052    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-kube-api-access-6gptr" pod "a551643a-4108-4d0d-be08-6add34edffb1" (UID: "a551643a-4108-4d0d-be08-6add34edffb1"). InnerVolumeSpecName "kube-api-access-6gptr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:40.147423 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.147402    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-istio-token" pod "a551643a-4108-4d0d-be08-6add34edffb1" (UID: "a551643a-4108-4d0d-be08-6add34edffb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:40.148491 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.148465    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-envoy" pod "a551643a-4108-4d0d-be08-6add34edffb1" (UID: "a551643a-4108-4d0d-be08-6add34edffb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:40.149657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.149634    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a551643a-4108-4d0d-be08-6add34edffb1-istio-podinfo" pod "a551643a-4108-4d0d-be08-6add34edffb1" (UID: "a551643a-4108-4d0d-be08-6add34edffb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:40.213787 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.213736    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"]
Oct 29 22:59:40.260128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.251755    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.260128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.251789    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a551643a-4108-4d0d-be08-6add34edffb1-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.260128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.251803    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.260128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.251815    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a551643a-4108-4d0d-be08-6add34edffb1-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.260128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.251828    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gptr\" (UniqueName: \"kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-kube-api-access-6gptr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.260128 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.251843    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a551643a-4108-4d0d-be08-6add34edffb1-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:40.349025 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.348930    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:40.477969 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.477934    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:40.834140 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.834105    4860 generic.go:356] "Generic (PLEG): container finished" podID="a551643a-4108-4d0d-be08-6add34edffb1" containerID="56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3" exitCode=0
Oct 29 22:59:40.834383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.834154    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"
Oct 29 22:59:40.834383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.834194    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" event={"ID":"a551643a-4108-4d0d-be08-6add34edffb1","Type":"ContainerDied","Data":"56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3"}
Oct 29 22:59:40.834383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.834253    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw" event={"ID":"a551643a-4108-4d0d-be08-6add34edffb1","Type":"ContainerDied","Data":"0b8061cab42b3069f6f5b00ead23b623126a572e5fa7174102346574942ee2f0"}
Oct 29 22:59:40.834383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.834277    4860 scope.go:122] "RemoveContainer" containerID="56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3"
Oct 29 22:59:40.847841 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.847824    4860 scope.go:122] "RemoveContainer" containerID="3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386"
Oct 29 22:59:40.858622 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.858604    4860 scope.go:122] "RemoveContainer" containerID="56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3"
Oct 29 22:59:40.858929 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:59:40.858909    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3\": container with ID starting with 56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3 not found: ID does not exist" containerID="56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3"
Oct 29 22:59:40.858990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.858936    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3"} err="failed to get container status \"56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3\": rpc error: code = NotFound desc = could not find container \"56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3\": container with ID starting with 56bc79dfacbab557e02ea89c552056bc004d84940fd65ef5d13c7517dd388dc3 not found: ID does not exist"
Oct 29 22:59:40.858990 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.858954    4860 scope.go:122] "RemoveContainer" containerID="3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386"
Oct 29 22:59:40.859227 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:59:40.859205    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386\": container with ID starting with 3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386 not found: ID does not exist" containerID="3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386"
Oct 29 22:59:40.859270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.859234    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386"} err="failed to get container status \"3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386\": rpc error: code = NotFound desc = could not find container \"3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386\": container with ID starting with 3b24ed12b3caad950175847dec683624efd4aa4d1e213736bceb09813ec60386 not found: ID does not exist"
Oct 29 22:59:40.910735 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.910709    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"]
Oct 29 22:59:40.922795 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:40.922771    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-db7419-openshift-default-9dd55d5bc-q2xvw"]
Oct 29 22:59:42.122837 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:42.122780    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a551643a-4108-4d0d-be08-6add34edffb1" path="/var/lib/kubelet/pods/a551643a-4108-4d0d-be08-6add34edffb1/volumes"
Oct 29 22:59:42.259125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:42.259086    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:42.432270 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:42.432241    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:42.654272 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:42.654242    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:42.868755 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:42.868665    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m" podUID="d55a1f4c-beeb-4c8d-bccb-d07d86dda333" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:42.868922 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:42.868795    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72" podUID="2c8afc25-6b5d-4136-8b69-0ef94a9ae866" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:44.005632 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:44.005590    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7" podUID="90c5bd3e-fea2-497c-9e35-2d0c5a0077c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:44.295507 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:44.295426    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:45.866928 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:45.866897    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"]
Oct 29 22:59:45.902821 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:45.902790    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"]
Oct 29 22:59:46.027383 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:46.027348    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:46.164224 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:46.164133    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:46.613936 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:46.613902    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:48.059350 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:59:48.059302    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 22:59:48.232618 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:48.232589    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:49.485157 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.485132    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"
Oct 29 22:59:49.602338 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.602292    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-kube-api-access-x28zb\" (UniqueName: \"kubernetes.io/projected/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-kube-api-access-x28zb\") pod \"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9\" (UID: \"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9\") "
Oct 29 22:59:49.602523 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.602362    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-plugins-config-volume\") pod \"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9\" (UID: \"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9\") "
Oct 29 22:59:49.602761 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.602736    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-plugins-config-volume" pod "90c5bd3e-fea2-497c-9e35-2d0c5a0077c9" (UID: "90c5bd3e-fea2-497c-9e35-2d0c5a0077c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:49.603077 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.603057    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:49.604600 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.604577    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-kube-api-access-x28zb" pod "90c5bd3e-fea2-497c-9e35-2d0c5a0077c9" (UID: "90c5bd3e-fea2-497c-9e35-2d0c5a0077c9"). InnerVolumeSpecName "kube-api-access-x28zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:49.705116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.705048    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x28zb\" (UniqueName: \"kubernetes.io/projected/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9-kube-api-access-x28zb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:49.875288 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.875237    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:49.969204 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.969099    4860 generic.go:356] "Generic (PLEG): container finished" podID="90c5bd3e-fea2-497c-9e35-2d0c5a0077c9" containerID="726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801" exitCode=0
Oct 29 22:59:49.969604 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.969572    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7" event={"ID":"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9","Type":"ContainerDied","Data":"726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801"}
Oct 29 22:59:49.969806 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.969788    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7" event={"ID":"90c5bd3e-fea2-497c-9e35-2d0c5a0077c9","Type":"ContainerDied","Data":"6d19d8dac31eee50f7038d4a2162a7eca0c4d0a20af533aa3681517d258c0f78"}
Oct 29 22:59:49.969974 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.969959    4860 scope.go:122] "RemoveContainer" containerID="726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801"
Oct 29 22:59:49.970196 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.969684    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"
Oct 29 22:59:49.987246 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.987218    4860 scope.go:122] "RemoveContainer" containerID="726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801"
Oct 29 22:59:49.987555 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:59:49.987533    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801\": container with ID starting with 726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801 not found: ID does not exist" containerID="726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801"
Oct 29 22:59:49.987678 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:49.987564    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801"} err="failed to get container status \"726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801\": rpc error: code = NotFound desc = could not find container \"726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801\": container with ID starting with 726786f9280cdb410429930ec741fcc9e650431f5fc3004a9ec50b1d2cd59801 not found: ID does not exist"
Oct 29 22:59:50.031022 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:50.030992    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"]
Oct 29 22:59:50.043850 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:50.043827    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aaa3c6-845674cd68-8fgr7"]
Oct 29 22:59:50.116812 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:50.116773    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c5bd3e-fea2-497c-9e35-2d0c5a0077c9" path="/var/lib/kubelet/pods/90c5bd3e-fea2-497c-9e35-2d0c5a0077c9/volumes"
Oct 29 22:59:50.867983 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:50.867941    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx" podUID="f5e951ea-8eb5-4a3a-aabf-dde45406c921" containerName="payload-processing" containerID="cri-o://c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0" gracePeriod=25
Oct 29 22:59:50.903374 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:50.903291    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw" podUID="4aaed535-f4bd-430d-8528-af2feb5a8ca4" containerName="payload-pre-processing" containerID="cri-o://9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc" gracePeriod=25
Oct 29 22:59:51.366466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:51.366433    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:52.647056 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:52.647014    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw" podUID="4aaed535-f4bd-430d-8528-af2feb5a8ca4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:52.647466 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:52.647063    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx" podUID="f5e951ea-8eb5-4a3a-aabf-dde45406c921" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:52.867265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:52.867223    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m" podUID="d55a1f4c-beeb-4c8d-bccb-d07d86dda333" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:52.867265 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:52.867256    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72" podUID="2c8afc25-6b5d-4136-8b69-0ef94a9ae866" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:59:53.034365 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:53.034309    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:54.505915 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:54.505882    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:55.261116 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.261081    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main" containerID="cri-o://4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42" gracePeriod=45
Oct 29 22:59:55.553083 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.553052    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:59:55.618808 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.618775    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-model-cache\") pod \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") "
Oct 29 22:59:55.618956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.618832    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tmp-dir\") pod \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") "
Oct 29 22:59:55.618956 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.618924    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-dshm\") pod \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") "
Oct 29 22:59:55.619082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.619024    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tmp-dir" pod "d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" (UID: "d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:55.619082 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.619054    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-model-cache" pod "d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" (UID: "d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:55.619203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.619099    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-kube-api-access-jqmfc\" (UniqueName: \"kubernetes.io/projected/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-kube-api-access-jqmfc\") pod \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") "
Oct 29 22:59:55.619203 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.619133    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tls-certs\") pod \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") "
Oct 29 22:59:55.619369 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.619256    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-home\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-home\") pod \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\" (UID: \"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d\") "
Oct 29 22:59:55.620112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.619527    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-home" pod "d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" (UID: "d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:55.620112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.620059    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:55.620112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.620079    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:55.620112 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.620091    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:55.621279 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.621258    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-dshm" pod "d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" (UID: "d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:55.621474 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.621454    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-kube-api-access-jqmfc" pod "d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" (UID: "d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d"). InnerVolumeSpecName "kube-api-access-jqmfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:55.621828 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.621804    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tls-certs" pod "d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" (UID: "d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:55.721262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.721230    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:55.721262 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.721261    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmfc\" (UniqueName: \"kubernetes.io/projected/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-kube-api-access-jqmfc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:55.721501 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:55.721272    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.045526 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.045491    4860 generic.go:356] "Generic (PLEG): container finished" podID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerID="4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42" exitCode=0
Oct 29 22:59:56.045716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.045542    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"
Oct 29 22:59:56.045716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.045546    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" event={"ID":"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d","Type":"ContainerDied","Data":"4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42"}
Oct 29 22:59:56.045716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.045684    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx" event={"ID":"d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d","Type":"ContainerDied","Data":"2b86168baaa9c5a62293a819f55e404202bdc6526bdf9d8b46e1d424d1ad60c0"}
Oct 29 22:59:56.045716 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.045712    4860 scope.go:122] "RemoveContainer" containerID="4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42"
Oct 29 22:59:56.064351 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.063452    4860 scope.go:122] "RemoveContainer" containerID="4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42"
Oct 29 22:59:56.064869 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:59:56.064842    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42\": container with ID starting with 4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42 not found: ID does not exist" containerID="4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42"
Oct 29 22:59:56.064975 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.064878    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42"} err="failed to get container status \"4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42\": rpc error: code = NotFound desc = could not find container \"4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42\": container with ID starting with 4fe25669534cbfc3121d91dad6df800bdb0ef8927d974591d469e73034320a42 not found: ID does not exist"
Oct 29 22:59:56.079380 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.079314    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:56.168344 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.167838    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"]
Oct 29 22:59:56.177856 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.177825    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-784e7a/simulator-w6-784e7a-kserve-6f9d798769-glghx"]
Oct 29 22:59:56.812295 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.812224    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"
Oct 29 22:59:56.861111 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.861086    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"
Oct 29 22:59:56.963587 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.963543    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-plugins-config-volume\") pod \"2c8afc25-6b5d-4136-8b69-0ef94a9ae866\" (UID: \"2c8afc25-6b5d-4136-8b69-0ef94a9ae866\") "
Oct 29 22:59:56.963753 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.963704    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-kube-api-access-tdr94\" (UniqueName: \"kubernetes.io/projected/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-kube-api-access-tdr94\") pod \"2c8afc25-6b5d-4136-8b69-0ef94a9ae866\" (UID: \"2c8afc25-6b5d-4136-8b69-0ef94a9ae866\") "
Oct 29 22:59:56.963869 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.963851    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-kube-api-access-gwv96\" (UniqueName: \"kubernetes.io/projected/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-kube-api-access-gwv96\") pod \"d55a1f4c-beeb-4c8d-bccb-d07d86dda333\" (UID: \"d55a1f4c-beeb-4c8d-bccb-d07d86dda333\") "
Oct 29 22:59:56.963992 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.963970    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-plugins-config-volume" pod "2c8afc25-6b5d-4136-8b69-0ef94a9ae866" (UID: "2c8afc25-6b5d-4136-8b69-0ef94a9ae866"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:56.964089 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.964043    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-plugins-config-volume\") pod \"d55a1f4c-beeb-4c8d-bccb-d07d86dda333\" (UID: \"d55a1f4c-beeb-4c8d-bccb-d07d86dda333\") "
Oct 29 22:59:56.964541 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.964514    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-plugins-config-volume" pod "d55a1f4c-beeb-4c8d-bccb-d07d86dda333" (UID: "d55a1f4c-beeb-4c8d-bccb-d07d86dda333"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:56.964744 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.964726    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.964835 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.964746    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:56.966154 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.966123    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-kube-api-access-gwv96" pod "d55a1f4c-beeb-4c8d-bccb-d07d86dda333" (UID: "d55a1f4c-beeb-4c8d-bccb-d07d86dda333"). InnerVolumeSpecName "kube-api-access-gwv96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:56.966642 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:56.966620    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-kube-api-access-tdr94" pod "2c8afc25-6b5d-4136-8b69-0ef94a9ae866" (UID: "2c8afc25-6b5d-4136-8b69-0ef94a9ae866"). InnerVolumeSpecName "kube-api-access-tdr94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:57.056886 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.056852    4860 generic.go:356] "Generic (PLEG): container finished" podID="d55a1f4c-beeb-4c8d-bccb-d07d86dda333" containerID="99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4" exitCode=0
Oct 29 22:59:57.057019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.056895    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m" event={"ID":"d55a1f4c-beeb-4c8d-bccb-d07d86dda333","Type":"ContainerDied","Data":"99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4"}
Oct 29 22:59:57.057019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.056918    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"
Oct 29 22:59:57.057019 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.057001    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m" event={"ID":"d55a1f4c-beeb-4c8d-bccb-d07d86dda333","Type":"ContainerDied","Data":"1444578c6d680b549398aee2b1fdd5c64e41cdda548c9a71bb6453e6d016aca2"}
Oct 29 22:59:57.057200 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.057032    4860 scope.go:122] "RemoveContainer" containerID="99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4"
Oct 29 22:59:57.058657 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.058634    4860 generic.go:356] "Generic (PLEG): container finished" podID="2c8afc25-6b5d-4136-8b69-0ef94a9ae866" containerID="94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086" exitCode=0
Oct 29 22:59:57.058766 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.058705    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"
Oct 29 22:59:57.058986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.058959    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72" event={"ID":"2c8afc25-6b5d-4136-8b69-0ef94a9ae866","Type":"ContainerDied","Data":"94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086"}
Oct 29 22:59:57.059109 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.059005    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72" event={"ID":"2c8afc25-6b5d-4136-8b69-0ef94a9ae866","Type":"ContainerDied","Data":"75ddc4f9a0f65913a38b260ed246ea76232d3e7a9919df5ba1601858b1c9449a"}
Oct 29 22:59:57.068401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.067720    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdr94\" (UniqueName: \"kubernetes.io/projected/2c8afc25-6b5d-4136-8b69-0ef94a9ae866-kube-api-access-tdr94\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:57.068401 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.067750    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwv96\" (UniqueName: \"kubernetes.io/projected/d55a1f4c-beeb-4c8d-bccb-d07d86dda333-kube-api-access-gwv96\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 22:59:57.071125 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.071106    4860 scope.go:122] "RemoveContainer" containerID="99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4"
Oct 29 22:59:57.071496 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:59:57.071466    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4\": container with ID starting with 99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4 not found: ID does not exist" containerID="99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4"
Oct 29 22:59:57.071594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.071509    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4"} err="failed to get container status \"99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4\": rpc error: code = NotFound desc = could not find container \"99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4\": container with ID starting with 99c309e0c6b930c92e660584c3972dfe20219c717f785bc424045ddadb9e90e4 not found: ID does not exist"
Oct 29 22:59:57.071594 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.071534    4860 scope.go:122] "RemoveContainer" containerID="94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086"
Oct 29 22:59:57.082428 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.082406    4860 scope.go:122] "RemoveContainer" containerID="94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086"
Oct 29 22:59:57.082693 ip-10-0-128-31 kubenswrapper[4860]: E1029 22:59:57.082674    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086\": container with ID starting with 94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086 not found: ID does not exist" containerID="94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086"
Oct 29 22:59:57.082767 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.082709    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086"} err="failed to get container status \"94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086\": rpc error: code = NotFound desc = could not find container \"94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086\": container with ID starting with 94030ce845030d63d2c084404d7fd7f63c0061470f97e05e813ef02d28445086 not found: ID does not exist"
Oct 29 22:59:57.118292 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.118256    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"]
Oct 29 22:59:57.123420 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.123398    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f8ca96-576b496c86-rkx2m"]
Oct 29 22:59:57.160914 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.160890    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"]
Oct 29 22:59:57.165467 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.165433    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f8ca96-696c58bc6c-4dw72"]
Oct 29 22:59:57.740986 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:57.740956    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:58.115416 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:58.115386    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8afc25-6b5d-4136-8b69-0ef94a9ae866" path="/var/lib/kubelet/pods/2c8afc25-6b5d-4136-8b69-0ef94a9ae866/volumes"
Oct 29 22:59:58.115842 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:58.115826    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d55a1f4c-beeb-4c8d-bccb-d07d86dda333" path="/var/lib/kubelet/pods/d55a1f4c-beeb-4c8d-bccb-d07d86dda333/volumes"
Oct 29 22:59:58.116257 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:58.116243    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" path="/var/lib/kubelet/pods/d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d/volumes"
Oct 29 22:59:59.015516 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:59.015481    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:59.617348 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:59.617297    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:59.876080 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:59.875979    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"]
Oct 29 22:59:59.886453 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:59.886429    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f8ca96-7dt8m"]
Oct 29 22:59:59.896620 ip-10-0-128-31 kubenswrapper[4860]: I1029 22:59:59.896593    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:00.027348 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.022823    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:00.116762 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.116725    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bbf7823-986b-41d2-8904-8c5e66811426" path="/var/lib/kubelet/pods/9bbf7823-986b-41d2-8904-8c5e66811426/volumes"
Oct 29 23:00:00.147369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.147275    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"]
Oct 29 23:00:00.148387 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.148316    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy"
Oct 29 23:00:00.148669 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.148591    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy"
Oct 29 23:00:00.148806 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.148713    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d55a1f4c-beeb-4c8d-bccb-d07d86dda333" containerName="payload-processing"
Oct 29 23:00:00.148806 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.148723    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55a1f4c-beeb-4c8d-bccb-d07d86dda333" containerName="payload-processing"
Oct 29 23:00:00.148806 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.148761    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main"
Oct 29 23:00:00.148806 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.148770    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main"
Oct 29 23:00:00.148806 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.148796    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2c8afc25-6b5d-4136-8b69-0ef94a9ae866" containerName="payload-pre-processing"
Oct 29 23:00:00.148806 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.148804    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8afc25-6b5d-4136-8b69-0ef94a9ae866" containerName="payload-pre-processing"
Oct 29 23:00:00.149125 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.148837    4860 cpu_manager.go:414] "containerMap: removing container" podUID="90c5bd3e-fea2-497c-9e35-2d0c5a0077c9" containerName="payload-processing"
Oct 29 23:00:00.149125 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.148845    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c5bd3e-fea2-497c-9e35-2d0c5a0077c9" containerName="payload-processing"
Oct 29 23:00:00.149125 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.149049    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c5bd3e-fea2-497c-9e35-2d0c5a0077c9" containerName="payload-processing"
Oct 29 23:00:00.149125 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.149065    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy"
Oct 29 23:00:00.149125 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.149081    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d55a1f4c-beeb-4c8d-bccb-d07d86dda333" containerName="payload-processing"
Oct 29 23:00:00.149125 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.149096    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e09943-de5c-4a6d-8cf0-5ebd6ac32f0d" containerName="main"
Oct 29 23:00:00.149125 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.149109    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8afc25-6b5d-4136-8b69-0ef94a9ae866" containerName="payload-pre-processing"
Oct 29 23:00:00.155274 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.155252    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"]
Oct 29 23:00:00.155416 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.155384    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"
Oct 29 23:00:00.155993 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.155972    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy"
Oct 29 23:00:00.155993 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.155992    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy"
Oct 29 23:00:00.156226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.156210    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a551643a-4108-4d0d-be08-6add34edffb1" containerName="istio-proxy"
Oct 29 23:00:00.160248 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.160230    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"]
Oct 29 23:00:00.160388 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.160373    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"
Oct 29 23:00:00.168189 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.168140    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"
Oct 29 23:00:00.182475 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.176045    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"]
Oct 29 23:00:00.188088 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.188055    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:00:00.197594 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.195407    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"]
Oct 29 23:00:00.238705 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.238643    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"]
Oct 29 23:00:00.239200 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.238764    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"
Oct 29 23:00:00.248449 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.247689    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86\" (UID: \"2f461145-f50b-4ebd-b9fa-25edb7d2af1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"
Oct 29 23:00:00.248449 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.247775    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfv7v\" (UniqueName: \"kubernetes.io/projected/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-kube-api-access-jfv7v\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4\" (UID: \"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"
Oct 29 23:00:00.248449 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.247837    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzw48\" (UniqueName: \"kubernetes.io/projected/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-kube-api-access-lzw48\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt\" (UID: \"cb80ca82-acc6-4bdf-b76a-f39eb983f49a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"
Oct 29 23:00:00.248449 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.247961    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt\" (UID: \"cb80ca82-acc6-4bdf-b76a-f39eb983f49a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"
Oct 29 23:00:00.248449 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.248042    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4\" (UID: \"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"
Oct 29 23:00:00.248449 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.248108    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk56g\" (UniqueName: \"kubernetes.io/projected/2247bdec-f506-4144-a05a-884528c5c132-kube-api-access-hk56g\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk\" (UID: \"2247bdec-f506-4144-a05a-884528c5c132\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"
Oct 29 23:00:00.248449 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.248223    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2247bdec-f506-4144-a05a-884528c5c132-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk\" (UID: \"2247bdec-f506-4144-a05a-884528c5c132\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"
Oct 29 23:00:00.248449 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.248254    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76pns\" (UniqueName: \"kubernetes.io/projected/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-kube-api-access-76pns\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86\" (UID: \"2f461145-f50b-4ebd-b9fa-25edb7d2af1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"
Oct 29 23:00:00.265139 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.245834    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"]
Oct 29 23:00:00.274560 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.274535    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"]
Oct 29 23:00:00.274786 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.274768    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"]
Oct 29 23:00:00.277519 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.276886    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"
Oct 29 23:00:00.282545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.282516    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"]
Oct 29 23:00:00.282678 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.282625    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"
Oct 29 23:00:00.294498 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.294474    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"]
Oct 29 23:00:00.301814 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.301794    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"]
Oct 29 23:00:00.311192 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.311170    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"]
Oct 29 23:00:00.317856 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.317822    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"
Oct 29 23:00:00.329555 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.329533    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"
Oct 29 23:00:00.347864 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.347833    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"]
Oct 29 23:00:00.350281 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.350234    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4\" (UID: \"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"
Oct 29 23:00:00.350431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.350369    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk56g\" (UniqueName: \"kubernetes.io/projected/2247bdec-f506-4144-a05a-884528c5c132-kube-api-access-hk56g\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk\" (UID: \"2247bdec-f506-4144-a05a-884528c5c132\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"
Oct 29 23:00:00.350542 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.350524    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2247bdec-f506-4144-a05a-884528c5c132-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk\" (UID: \"2247bdec-f506-4144-a05a-884528c5c132\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"
Oct 29 23:00:00.350614 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.350577    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76pns\" (UniqueName: \"kubernetes.io/projected/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-kube-api-access-76pns\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86\" (UID: \"2f461145-f50b-4ebd-b9fa-25edb7d2af1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"
Oct 29 23:00:00.350683 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.350617    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86\" (UID: \"2f461145-f50b-4ebd-b9fa-25edb7d2af1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"
Oct 29 23:00:00.350743 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.350721    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfv7v\" (UniqueName: \"kubernetes.io/projected/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-kube-api-access-jfv7v\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4\" (UID: \"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"
Oct 29 23:00:00.350818 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.350797    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzw48\" (UniqueName: \"kubernetes.io/projected/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-kube-api-access-lzw48\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt\" (UID: \"cb80ca82-acc6-4bdf-b76a-f39eb983f49a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"
Oct 29 23:00:00.350960 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.350931    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt\" (UID: \"cb80ca82-acc6-4bdf-b76a-f39eb983f49a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"
Oct 29 23:00:00.351914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.351891    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt\" (UID: \"cb80ca82-acc6-4bdf-b76a-f39eb983f49a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"
Oct 29 23:00:00.352591 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.352566    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4\" (UID: \"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"
Oct 29 23:00:00.353542 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.353519    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2247bdec-f506-4144-a05a-884528c5c132-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk\" (UID: \"2247bdec-f506-4144-a05a-884528c5c132\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"
Oct 29 23:00:00.354110 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.354085    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86\" (UID: \"2f461145-f50b-4ebd-b9fa-25edb7d2af1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"
Oct 29 23:00:00.355772 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.355725    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"]
Oct 29 23:00:00.359044 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.359021    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:00.364396 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.364377    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"]
Oct 29 23:00:00.364628 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.364609    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"
Oct 29 23:00:00.370112 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.370093    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"
Oct 29 23:00:00.371035 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.371012    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"]
Oct 29 23:00:00.375968 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.375943    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk56g\" (UniqueName: \"kubernetes.io/projected/2247bdec-f506-4144-a05a-884528c5c132-kube-api-access-hk56g\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk\" (UID: \"2247bdec-f506-4144-a05a-884528c5c132\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"
Oct 29 23:00:00.376488 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.376460    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzw48\" (UniqueName: \"kubernetes.io/projected/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-kube-api-access-lzw48\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt\" (UID: \"cb80ca82-acc6-4bdf-b76a-f39eb983f49a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"
Oct 29 23:00:00.376723 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.376702    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76pns\" (UniqueName: \"kubernetes.io/projected/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-kube-api-access-76pns\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86\" (UID: \"2f461145-f50b-4ebd-b9fa-25edb7d2af1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"
Oct 29 23:00:00.381372 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.378107    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfv7v\" (UniqueName: \"kubernetes.io/projected/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-kube-api-access-jfv7v\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4\" (UID: \"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"
Oct 29 23:00:00.381372 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.378299    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"]
Oct 29 23:00:00.383590 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.383566    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"]
Oct 29 23:00:00.390351 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.390313    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"]
Oct 29 23:00:00.390534 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.390404    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"
Oct 29 23:00:00.399446 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.399379    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"]
Oct 29 23:00:00.407196 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.407176    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"]
Oct 29 23:00:00.407304 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.407269    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"
Oct 29 23:00:00.412623 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.412605    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"]
Oct 29 23:00:00.416889 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.416870    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"]
Oct 29 23:00:00.452377 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.452346    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/521dba0b-944b-4ee0-8e00-60fa687c94bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl\" (UID: \"521dba0b-944b-4ee0-8e00-60fa687c94bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"
Oct 29 23:00:00.452527 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.452411    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b151b922-7435-4d4e-a403-3c911de5d8f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k\" (UID: \"b151b922-7435-4d4e-a403-3c911de5d8f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"
Oct 29 23:00:00.452590 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.452527    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt6m8\" (UniqueName: \"kubernetes.io/projected/521dba0b-944b-4ee0-8e00-60fa687c94bd-kube-api-access-xt6m8\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl\" (UID: \"521dba0b-944b-4ee0-8e00-60fa687c94bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"
Oct 29 23:00:00.452646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.452617    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a942f-a12b-42b1-8b11-b6e2c5103798-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n\" (UID: \"7c7a942f-a12b-42b1-8b11-b6e2c5103798\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"
Oct 29 23:00:00.452726 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.452708    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whxmd\" (UniqueName: \"kubernetes.io/projected/b151b922-7435-4d4e-a403-3c911de5d8f6-kube-api-access-whxmd\") pod \"maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k\" (UID: \"b151b922-7435-4d4e-a403-3c911de5d8f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"
Oct 29 23:00:00.452797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.452750    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32dbf81d-0294-4922-b785-1a0b3950017a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd\" (UID: \"32dbf81d-0294-4922-b785-1a0b3950017a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"
Oct 29 23:00:00.452797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.452786    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmnbf\" (UniqueName: \"kubernetes.io/projected/32dbf81d-0294-4922-b785-1a0b3950017a-kube-api-access-vmnbf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd\" (UID: \"32dbf81d-0294-4922-b785-1a0b3950017a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"
Oct 29 23:00:00.452965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.452945    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e276a4c-e944-495d-8d4e-f80ebd436f05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds\" (UID: \"4e276a4c-e944-495d-8d4e-f80ebd436f05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"
Oct 29 23:00:00.453104 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.453082    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvmtz\" (UniqueName: \"kubernetes.io/projected/4e276a4c-e944-495d-8d4e-f80ebd436f05-kube-api-access-nvmtz\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds\" (UID: \"4e276a4c-e944-495d-8d4e-f80ebd436f05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"
Oct 29 23:00:00.453179 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.453124    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2v7s\" (UniqueName: \"kubernetes.io/projected/7c7a942f-a12b-42b1-8b11-b6e2c5103798-kube-api-access-g2v7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n\" (UID: \"7c7a942f-a12b-42b1-8b11-b6e2c5103798\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"
Oct 29 23:00:00.453179 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.453169    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7k7m\" (UniqueName: \"kubernetes.io/projected/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-kube-api-access-g7k7m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6\" (UID: \"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"
Oct 29 23:00:00.453423 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.453290    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6\" (UID: \"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"
Oct 29 23:00:00.479113 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.479088    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"
Oct 29 23:00:00.544353 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.540813    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"
Oct 29 23:00:00.585519 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.585242    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"
Oct 29 23:00:00.586368 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.586007    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a942f-a12b-42b1-8b11-b6e2c5103798-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n\" (UID: \"7c7a942f-a12b-42b1-8b11-b6e2c5103798\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"
Oct 29 23:00:00.586368 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.586078    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whxmd\" (UniqueName: \"kubernetes.io/projected/b151b922-7435-4d4e-a403-3c911de5d8f6-kube-api-access-whxmd\") pod \"maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k\" (UID: \"b151b922-7435-4d4e-a403-3c911de5d8f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"
Oct 29 23:00:00.586368 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.586264    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32dbf81d-0294-4922-b785-1a0b3950017a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd\" (UID: \"32dbf81d-0294-4922-b785-1a0b3950017a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"
Oct 29 23:00:00.586731 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.586376    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmnbf\" (UniqueName: \"kubernetes.io/projected/32dbf81d-0294-4922-b785-1a0b3950017a-kube-api-access-vmnbf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd\" (UID: \"32dbf81d-0294-4922-b785-1a0b3950017a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"
Oct 29 23:00:00.586936 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.586907    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e276a4c-e944-495d-8d4e-f80ebd436f05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds\" (UID: \"4e276a4c-e944-495d-8d4e-f80ebd436f05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"
Oct 29 23:00:00.587090 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.587056    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks\" (UID: \"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"
Oct 29 23:00:00.587177 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.587089    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32dbf81d-0294-4922-b785-1a0b3950017a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd\" (UID: \"32dbf81d-0294-4922-b785-1a0b3950017a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"
Oct 29 23:00:00.587177 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.587130    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8lnj\" (UniqueName: \"kubernetes.io/projected/0b239203-2773-4ff8-bc28-604940a34500-kube-api-access-r8lnj\") pod \"maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb\" (UID: \"0b239203-2773-4ff8-bc28-604940a34500\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"
Oct 29 23:00:00.587304 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.587251    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvmtz\" (UniqueName: \"kubernetes.io/projected/4e276a4c-e944-495d-8d4e-f80ebd436f05-kube-api-access-nvmtz\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds\" (UID: \"4e276a4c-e944-495d-8d4e-f80ebd436f05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"
Oct 29 23:00:00.587304 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.587287    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2v7s\" (UniqueName: \"kubernetes.io/projected/7c7a942f-a12b-42b1-8b11-b6e2c5103798-kube-api-access-g2v7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n\" (UID: \"7c7a942f-a12b-42b1-8b11-b6e2c5103798\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"
Oct 29 23:00:00.587477 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.587312    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b239203-2773-4ff8-bc28-604940a34500-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb\" (UID: \"0b239203-2773-4ff8-bc28-604940a34500\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"
Oct 29 23:00:00.587477 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.587384    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7k7m\" (UniqueName: \"kubernetes.io/projected/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-kube-api-access-g7k7m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6\" (UID: \"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"
Oct 29 23:00:00.587826 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.587800    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e276a4c-e944-495d-8d4e-f80ebd436f05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds\" (UID: \"4e276a4c-e944-495d-8d4e-f80ebd436f05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"
Oct 29 23:00:00.587953 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.587934    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6\" (UID: \"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"
Oct 29 23:00:00.588341 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.588301    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/521dba0b-944b-4ee0-8e00-60fa687c94bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl\" (UID: \"521dba0b-944b-4ee0-8e00-60fa687c94bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"
Oct 29 23:00:00.588453 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.588425    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b151b922-7435-4d4e-a403-3c911de5d8f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k\" (UID: \"b151b922-7435-4d4e-a403-3c911de5d8f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"
Oct 29 23:00:00.588790 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.588484    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvzpb\" (UniqueName: \"kubernetes.io/projected/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-kube-api-access-tvzpb\") pod \"maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks\" (UID: \"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"
Oct 29 23:00:00.588790 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.588537    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt6m8\" (UniqueName: \"kubernetes.io/projected/521dba0b-944b-4ee0-8e00-60fa687c94bd-kube-api-access-xt6m8\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl\" (UID: \"521dba0b-944b-4ee0-8e00-60fa687c94bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"
Oct 29 23:00:00.589078 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.588898    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6\" (UID: \"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"
Oct 29 23:00:00.589582 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.589556    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a942f-a12b-42b1-8b11-b6e2c5103798-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n\" (UID: \"7c7a942f-a12b-42b1-8b11-b6e2c5103798\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"
Oct 29 23:00:00.589778 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.589753    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/521dba0b-944b-4ee0-8e00-60fa687c94bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl\" (UID: \"521dba0b-944b-4ee0-8e00-60fa687c94bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"
Oct 29 23:00:00.591663 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.591639    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b151b922-7435-4d4e-a403-3c911de5d8f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k\" (UID: \"b151b922-7435-4d4e-a403-3c911de5d8f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"
Oct 29 23:00:00.595713 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.595527    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"
Oct 29 23:00:00.612924 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.612896    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt6m8\" (UniqueName: \"kubernetes.io/projected/521dba0b-944b-4ee0-8e00-60fa687c94bd-kube-api-access-xt6m8\") pod \"maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl\" (UID: \"521dba0b-944b-4ee0-8e00-60fa687c94bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"
Oct 29 23:00:00.616660 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.616604    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2v7s\" (UniqueName: \"kubernetes.io/projected/7c7a942f-a12b-42b1-8b11-b6e2c5103798-kube-api-access-g2v7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n\" (UID: \"7c7a942f-a12b-42b1-8b11-b6e2c5103798\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"
Oct 29 23:00:00.617496 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.617450    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmnbf\" (UniqueName: \"kubernetes.io/projected/32dbf81d-0294-4922-b785-1a0b3950017a-kube-api-access-vmnbf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd\" (UID: \"32dbf81d-0294-4922-b785-1a0b3950017a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"
Oct 29 23:00:00.618598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.618573    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvmtz\" (UniqueName: \"kubernetes.io/projected/4e276a4c-e944-495d-8d4e-f80ebd436f05-kube-api-access-nvmtz\") pod \"maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds\" (UID: \"4e276a4c-e944-495d-8d4e-f80ebd436f05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"
Oct 29 23:00:00.623999 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.623975    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7k7m\" (UniqueName: \"kubernetes.io/projected/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-kube-api-access-g7k7m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6\" (UID: \"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"
Oct 29 23:00:00.641733 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.641703    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whxmd\" (UniqueName: \"kubernetes.io/projected/b151b922-7435-4d4e-a403-3c911de5d8f6-kube-api-access-whxmd\") pod \"maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k\" (UID: \"b151b922-7435-4d4e-a403-3c911de5d8f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"
Oct 29 23:00:00.666112 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.664763    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"
Oct 29 23:00:00.679665 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.679632    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"
Oct 29 23:00:00.693036 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.690654    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks\" (UID: \"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"
Oct 29 23:00:00.693036 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.690717    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lnj\" (UniqueName: \"kubernetes.io/projected/0b239203-2773-4ff8-bc28-604940a34500-kube-api-access-r8lnj\") pod \"maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb\" (UID: \"0b239203-2773-4ff8-bc28-604940a34500\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"
Oct 29 23:00:00.693036 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.690795    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b239203-2773-4ff8-bc28-604940a34500-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb\" (UID: \"0b239203-2773-4ff8-bc28-604940a34500\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"
Oct 29 23:00:00.693036 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.690969    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvzpb\" (UniqueName: \"kubernetes.io/projected/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-kube-api-access-tvzpb\") pod \"maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks\" (UID: \"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"
Oct 29 23:00:00.693036 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.692130    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks\" (UID: \"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"
Oct 29 23:00:00.693036 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.692996    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b239203-2773-4ff8-bc28-604940a34500-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb\" (UID: \"0b239203-2773-4ff8-bc28-604940a34500\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"
Oct 29 23:00:00.728911 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.728878    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvzpb\" (UniqueName: \"kubernetes.io/projected/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-kube-api-access-tvzpb\") pod \"maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks\" (UID: \"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"
Oct 29 23:00:00.732039 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.731857    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"
Oct 29 23:00:00.735038 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.735008    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8lnj\" (UniqueName: \"kubernetes.io/projected/0b239203-2773-4ff8-bc28-604940a34500-kube-api-access-r8lnj\") pod \"maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb\" (UID: \"0b239203-2773-4ff8-bc28-604940a34500\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"
Oct 29 23:00:00.736545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.736514    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"
Oct 29 23:00:00.757295 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.757263    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"
Oct 29 23:00:00.757700 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.757680    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"
Oct 29 23:00:00.763636 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.763609    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"
Oct 29 23:00:00.795921 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.795888    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"]
Oct 29 23:00:00.814790 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:00:00.814746    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8314b5_c567_4ebd_8a0b_d55cf6b3047d.slice/crio-e9bf2bcb78373c0c5c41c3a9f375fc8875f327d393e83228eedf07c2d9cadfd6 WatchSource:0}: Error finding container e9bf2bcb78373c0c5c41c3a9f375fc8875f327d393e83228eedf07c2d9cadfd6: Status 404 returned error can't find the container with id e9bf2bcb78373c0c5c41c3a9f375fc8875f327d393e83228eedf07c2d9cadfd6
Oct 29 23:00:00.912574 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.895862    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"]
Oct 29 23:00:00.928708 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:00.928445    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"
Oct 29 23:00:00.946755 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:00:00.946510    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2247bdec_f506_4144_a05a_884528c5c132.slice/crio-57fad2b7cda15b287ab5191393fd3fd10e6f09dd45bd1f483fe46116ed1d40c4 WatchSource:0}: Error finding container 57fad2b7cda15b287ab5191393fd3fd10e6f09dd45bd1f483fe46116ed1d40c4: Status 404 returned error can't find the container with id 57fad2b7cda15b287ab5191393fd3fd10e6f09dd45bd1f483fe46116ed1d40c4
Oct 29 23:00:01.121890 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.121664    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4" event={"ID":"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d","Type":"ContainerStarted","Data":"e9bf2bcb78373c0c5c41c3a9f375fc8875f327d393e83228eedf07c2d9cadfd6"}
Oct 29 23:00:01.136389 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.136319    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk" event={"ID":"2247bdec-f506-4144-a05a-884528c5c132","Type":"ContainerStarted","Data":"57fad2b7cda15b287ab5191393fd3fd10e6f09dd45bd1f483fe46116ed1d40c4"}
Oct 29 23:00:01.159907 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.159878    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"]
Oct 29 23:00:01.175131 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.175100    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"]
Oct 29 23:00:01.178950 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:00:01.178922    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f461145_f50b_4ebd_b9fa_25edb7d2af1a.slice/crio-fc66ad5a21e31a4618d8c52ead6ce167d176fe7960c39cbe4931a900e190ebd7 WatchSource:0}: Error finding container fc66ad5a21e31a4618d8c52ead6ce167d176fe7960c39cbe4931a900e190ebd7: Status 404 returned error can't find the container with id fc66ad5a21e31a4618d8c52ead6ce167d176fe7960c39cbe4931a900e190ebd7
Oct 29 23:00:01.190546 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:00:01.190524    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb80ca82_acc6_4bdf_b76a_f39eb983f49a.slice/crio-e01d8db7e89ace85ed3a736d46a305b46993aaee4b10dad8aaaeaf7fb7d52bac WatchSource:0}: Error finding container e01d8db7e89ace85ed3a736d46a305b46993aaee4b10dad8aaaeaf7fb7d52bac: Status 404 returned error can't find the container with id e01d8db7e89ace85ed3a736d46a305b46993aaee4b10dad8aaaeaf7fb7d52bac
Oct 29 23:00:01.274921 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.274879    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"]
Oct 29 23:00:01.277303 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:00:01.277276    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521dba0b_944b_4ee0_8e00_60fa687c94bd.slice/crio-e11cacb6731588fbfd847f22aa2c961fccb4628213b1ccda5f1c023ca67b7934 WatchSource:0}: Error finding container e11cacb6731588fbfd847f22aa2c961fccb4628213b1ccda5f1c023ca67b7934: Status 404 returned error can't find the container with id e11cacb6731588fbfd847f22aa2c961fccb4628213b1ccda5f1c023ca67b7934
Oct 29 23:00:01.281363 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.281337    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"]
Oct 29 23:00:01.374590 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:00:01.374557    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e276a4c_e944_495d_8d4e_f80ebd436f05.slice/crio-521e4df873de0192032eda112d1b235c2c6f6c035df1b81eb72aec7a5ed8e989 WatchSource:0}: Error finding container 521e4df873de0192032eda112d1b235c2c6f6c035df1b81eb72aec7a5ed8e989: Status 404 returned error can't find the container with id 521e4df873de0192032eda112d1b235c2c6f6c035df1b81eb72aec7a5ed8e989
Oct 29 23:00:01.375575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.375548    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"]
Oct 29 23:00:01.387864 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.387835    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:01.403012 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.402987    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"]
Oct 29 23:00:01.430779 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.430753    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"]
Oct 29 23:00:01.435952 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.435927    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"]
Oct 29 23:00:01.440647 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.440624    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"]
Oct 29 23:00:01.666883 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.665077    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:01.672034 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:01.671745    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"]
Oct 29 23:00:02.163592 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.163558    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"]
Oct 29 23:00:02.177994 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.177967    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"]
Oct 29 23:00:02.178225 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.178207    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"]
Oct 29 23:00:02.180291 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.179734    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"
Oct 29 23:00:02.186083 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.186065    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4" event={"ID":"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d","Type":"ContainerStarted","Data":"56c0a9aecb99c44017a2f0225353929bccdba1ec0b9d178d4455b039fe52ee96"}
Oct 29 23:00:02.186200 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.186185    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds" event={"ID":"4e276a4c-e944-495d-8d4e-f80ebd436f05","Type":"ContainerStarted","Data":"521e4df873de0192032eda112d1b235c2c6f6c035df1b81eb72aec7a5ed8e989"}
Oct 29 23:00:02.186268 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.186258    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6" event={"ID":"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76","Type":"ContainerStarted","Data":"b242a13d025b97920cc88199beef746cd16ad9f4f8c0b4df2317891e7789bbc2"}
Oct 29 23:00:02.186365 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.186351    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks" event={"ID":"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c","Type":"ContainerStarted","Data":"7f92267be92b6b367d8aac7098db31682674b6c0caf0df7a1e2f97269adbf1a3"}
Oct 29 23:00:02.186473 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.186458    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n" event={"ID":"7c7a942f-a12b-42b1-8b11-b6e2c5103798","Type":"ContainerStarted","Data":"d2e1b5f6de4eccd9dce83a527314a45d8b41e70290dd0f04041128366ea20748"}
Oct 29 23:00:02.186686 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.186669    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"
Oct 29 23:00:02.189168 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.188615    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl" event={"ID":"521dba0b-944b-4ee0-8e00-60fa687c94bd","Type":"ContainerStarted","Data":"e11cacb6731588fbfd847f22aa2c961fccb4628213b1ccda5f1c023ca67b7934"}
Oct 29 23:00:02.189420 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.189401    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"]
Oct 29 23:00:02.191060 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.191037    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k" event={"ID":"b151b922-7435-4d4e-a403-3c911de5d8f6","Type":"ContainerStarted","Data":"3ccced3def1396db9e64e3cb882a8255842a24d1af9959a9b0f12bb6b9dcb327"}
Oct 29 23:00:02.193498 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.193474    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb" event={"ID":"0b239203-2773-4ff8-bc28-604940a34500","Type":"ContainerStarted","Data":"30238ba32beea8b9a2a46ba022e604cd175571ad56c6e1b065fcaf384368bf64"}
Oct 29 23:00:02.195973 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.195951    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86" event={"ID":"2f461145-f50b-4ebd-b9fa-25edb7d2af1a","Type":"ContainerStarted","Data":"a27588bf8ed6b63ab08a8857aac9cc0a16c4756f3a97aa7974085eda3c786058"}
Oct 29 23:00:02.196092 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.195990    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86" event={"ID":"2f461145-f50b-4ebd-b9fa-25edb7d2af1a","Type":"ContainerStarted","Data":"fc66ad5a21e31a4618d8c52ead6ce167d176fe7960c39cbe4931a900e190ebd7"}
Oct 29 23:00:02.199097 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.199072    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt" event={"ID":"cb80ca82-acc6-4bdf-b76a-f39eb983f49a","Type":"ContainerStarted","Data":"e01d8db7e89ace85ed3a736d46a305b46993aaee4b10dad8aaaeaf7fb7d52bac"}
Oct 29 23:00:02.199582 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.199558    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-83f120\""
Oct 29 23:00:02.199682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.199645    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-83f120-dockercfg-fmhr8\""
Oct 29 23:00:02.201705 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.201682    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd" event={"ID":"32dbf81d-0294-4922-b785-1a0b3950017a","Type":"ContainerStarted","Data":"a0f0c5112311ca366c9e64e180a0591afd2dd0edca970412cd97bda2801159ad"}
Oct 29 23:00:02.222387 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.213070    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk" event={"ID":"2247bdec-f506-4144-a05a-884528c5c132","Type":"ContainerStarted","Data":"cfc58b1362ff99feacc8c7d4c66a353aadeca0ada2443765b0a10b1ce9910709"}
Oct 29 23:00:02.228758 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.228708    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4" podStartSLOduration=2.228689317 podStartE2EDuration="2.228689317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.20508698 +0000 UTC m=+2744.229863127" watchObservedRunningTime="2026-10-29 23:00:02.228689317 +0000 UTC m=+2744.253465448"
Oct 29 23:00:02.247616 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.247561    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sr8k\" (UniqueName: \"kubernetes.io/projected/130135b0-3ddc-47eb-a913-5648e0e93b8d-kube-api-access-9sr8k\") pod \"payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9\" (UID: \"130135b0-3ddc-47eb-a913-5648e0e93b8d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"
Oct 29 23:00:02.247616 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.247606    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk962\" (UniqueName: \"kubernetes.io/projected/9f4a5ad2-3f37-4854-975d-355793e94b71-kube-api-access-sk962\") pod \"payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5\" (UID: \"9f4a5ad2-3f37-4854-975d-355793e94b71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"
Oct 29 23:00:02.247799 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.247624    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130135b0-3ddc-47eb-a913-5648e0e93b8d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9\" (UID: \"130135b0-3ddc-47eb-a913-5648e0e93b8d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"
Oct 29 23:00:02.248210 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.248193    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f4a5ad2-3f37-4854-975d-355793e94b71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5\" (UID: \"9f4a5ad2-3f37-4854-975d-355793e94b71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"
Oct 29 23:00:02.259390 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.259335    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt" podStartSLOduration=2.259309725 podStartE2EDuration="2.259309725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.25429161 +0000 UTC m=+2744.279067737" watchObservedRunningTime="2026-10-29 23:00:02.259309725 +0000 UTC m=+2744.284085855"
Oct 29 23:00:02.342745 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.342687    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86" podStartSLOduration=2.342674291 podStartE2EDuration="2.342674291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.336816929 +0000 UTC m=+2744.361593081" watchObservedRunningTime="2026-10-29 23:00:02.342674291 +0000 UTC m=+2744.367450421"
Oct 29 23:00:02.350499 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.350464    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f4a5ad2-3f37-4854-975d-355793e94b71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5\" (UID: \"9f4a5ad2-3f37-4854-975d-355793e94b71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"
Oct 29 23:00:02.350637 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.350622    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sr8k\" (UniqueName: \"kubernetes.io/projected/130135b0-3ddc-47eb-a913-5648e0e93b8d-kube-api-access-9sr8k\") pod \"payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9\" (UID: \"130135b0-3ddc-47eb-a913-5648e0e93b8d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"
Oct 29 23:00:02.350717 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.350655    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk962\" (UniqueName: \"kubernetes.io/projected/9f4a5ad2-3f37-4854-975d-355793e94b71-kube-api-access-sk962\") pod \"payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5\" (UID: \"9f4a5ad2-3f37-4854-975d-355793e94b71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"
Oct 29 23:00:02.350717 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.350680    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130135b0-3ddc-47eb-a913-5648e0e93b8d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9\" (UID: \"130135b0-3ddc-47eb-a913-5648e0e93b8d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"
Oct 29 23:00:02.351296 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.351272    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f4a5ad2-3f37-4854-975d-355793e94b71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5\" (UID: \"9f4a5ad2-3f37-4854-975d-355793e94b71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"
Oct 29 23:00:02.351518 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.351493    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130135b0-3ddc-47eb-a913-5648e0e93b8d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9\" (UID: \"130135b0-3ddc-47eb-a913-5648e0e93b8d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"
Oct 29 23:00:02.376224 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.376193    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sr8k\" (UniqueName: \"kubernetes.io/projected/130135b0-3ddc-47eb-a913-5648e0e93b8d-kube-api-access-9sr8k\") pod \"payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9\" (UID: \"130135b0-3ddc-47eb-a913-5648e0e93b8d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"
Oct 29 23:00:02.384212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.384184    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk962\" (UniqueName: \"kubernetes.io/projected/9f4a5ad2-3f37-4854-975d-355793e94b71-kube-api-access-sk962\") pod \"payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5\" (UID: \"9f4a5ad2-3f37-4854-975d-355793e94b71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"
Oct 29 23:00:02.422921 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.422869    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk" podStartSLOduration=2.422845948 podStartE2EDuration="2.422845948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.418918155 +0000 UTC m=+2744.443694290" watchObservedRunningTime="2026-10-29 23:00:02.422845948 +0000 UTC m=+2744.447622082"
Oct 29 23:00:02.518823 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.518780    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:02.577177 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.576794    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"
Oct 29 23:00:02.597932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.593776    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"
Oct 29 23:00:02.647119 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.647092    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw" podUID="4aaed535-f4bd-430d-8528-af2feb5a8ca4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:02.648537 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.648478    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx" podUID="f5e951ea-8eb5-4a3a-aabf-dde45406c921" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:02.931784 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:00:02.931757    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4a5ad2_3f37_4854_975d_355793e94b71.slice/crio-3b768c4fde3060505119fbaf92e9da7b3256ba71eb1a2ec9496c47ea1527eb0a WatchSource:0}: Error finding container 3b768c4fde3060505119fbaf92e9da7b3256ba71eb1a2ec9496c47ea1527eb0a: Status 404 returned error can't find the container with id 3b768c4fde3060505119fbaf92e9da7b3256ba71eb1a2ec9496c47ea1527eb0a
Oct 29 23:00:02.933583 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.933561    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"]
Oct 29 23:00:02.961862 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:00:02.961834    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130135b0_3ddc_47eb_a913_5648e0e93b8d.slice/crio-dd6ce954dbcba5383f902993ac277c333b3b2b24c4ecf65dcb9550a7a003088c WatchSource:0}: Error finding container dd6ce954dbcba5383f902993ac277c333b3b2b24c4ecf65dcb9550a7a003088c: Status 404 returned error can't find the container with id dd6ce954dbcba5383f902993ac277c333b3b2b24c4ecf65dcb9550a7a003088c
Oct 29 23:00:02.973170 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:02.971380    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"]
Oct 29 23:00:03.226900 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.226870    4860 generic.go:356] "Generic (PLEG): container finished" podID="b151b922-7435-4d4e-a403-3c911de5d8f6" containerID="22c07c0178a9c09fc402bb5338b6c1200787132752c20bfc6c21f30cb4fba202" exitCode=60
Oct 29 23:00:03.227005 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.226964    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k" event={"ID":"b151b922-7435-4d4e-a403-3c911de5d8f6","Type":"ContainerDied","Data":"22c07c0178a9c09fc402bb5338b6c1200787132752c20bfc6c21f30cb4fba202"}
Oct 29 23:00:03.227502 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.227477    4860 scope.go:122] "RemoveContainer" containerID="22c07c0178a9c09fc402bb5338b6c1200787132752c20bfc6c21f30cb4fba202"
Oct 29 23:00:03.229214 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.229115    4860 generic.go:356] "Generic (PLEG): container finished" podID="0b239203-2773-4ff8-bc28-604940a34500" containerID="c42bef62738652eb5ab1f534aa4368c9f21b868e50cf0fa94da2305ebd3ff976" exitCode=60
Oct 29 23:00:03.229214 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.229190    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb" event={"ID":"0b239203-2773-4ff8-bc28-604940a34500","Type":"ContainerDied","Data":"c42bef62738652eb5ab1f534aa4368c9f21b868e50cf0fa94da2305ebd3ff976"}
Oct 29 23:00:03.229689 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.229628    4860 scope.go:122] "RemoveContainer" containerID="c42bef62738652eb5ab1f534aa4368c9f21b868e50cf0fa94da2305ebd3ff976"
Oct 29 23:00:03.231718 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.231695    4860 generic.go:356] "Generic (PLEG): container finished" podID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" containerID="a27588bf8ed6b63ab08a8857aac9cc0a16c4756f3a97aa7974085eda3c786058" exitCode=60
Oct 29 23:00:03.231826 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.231779    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86" event={"ID":"2f461145-f50b-4ebd-b9fa-25edb7d2af1a","Type":"ContainerDied","Data":"a27588bf8ed6b63ab08a8857aac9cc0a16c4756f3a97aa7974085eda3c786058"}
Oct 29 23:00:03.232148 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.232126    4860 scope.go:122] "RemoveContainer" containerID="a27588bf8ed6b63ab08a8857aac9cc0a16c4756f3a97aa7974085eda3c786058"
Oct 29 23:00:03.235255 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.235216    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9" event={"ID":"130135b0-3ddc-47eb-a913-5648e0e93b8d","Type":"ContainerStarted","Data":"bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf"}
Oct 29 23:00:03.235408 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.235279    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9" event={"ID":"130135b0-3ddc-47eb-a913-5648e0e93b8d","Type":"ContainerStarted","Data":"dd6ce954dbcba5383f902993ac277c333b3b2b24c4ecf65dcb9550a7a003088c"}
Oct 29 23:00:03.236003 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.235968    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"
Oct 29 23:00:03.240313 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.240157    4860 generic.go:356] "Generic (PLEG): container finished" podID="cb80ca82-acc6-4bdf-b76a-f39eb983f49a" containerID="81231caf2b89eb2ce7e623a5dfca7115eb780d4bcbfb74ec5a7d6efb506e7f95" exitCode=60
Oct 29 23:00:03.240313 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.240270    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt" event={"ID":"cb80ca82-acc6-4bdf-b76a-f39eb983f49a","Type":"ContainerDied","Data":"81231caf2b89eb2ce7e623a5dfca7115eb780d4bcbfb74ec5a7d6efb506e7f95"}
Oct 29 23:00:03.240511 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.240497    4860 scope.go:122] "RemoveContainer" containerID="81231caf2b89eb2ce7e623a5dfca7115eb780d4bcbfb74ec5a7d6efb506e7f95"
Oct 29 23:00:03.242400 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.242378    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd" event={"ID":"32dbf81d-0294-4922-b785-1a0b3950017a","Type":"ContainerStarted","Data":"4b06c41e2458d0c3d8c85166a9c8badf7bdedcc0671117dacabe3104c01808b0"}
Oct 29 23:00:03.247226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.247200    4860 generic.go:356] "Generic (PLEG): container finished" podID="2247bdec-f506-4144-a05a-884528c5c132" containerID="cfc58b1362ff99feacc8c7d4c66a353aadeca0ada2443765b0a10b1ce9910709" exitCode=60
Oct 29 23:00:03.247382 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.247320    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk" event={"ID":"2247bdec-f506-4144-a05a-884528c5c132","Type":"ContainerDied","Data":"cfc58b1362ff99feacc8c7d4c66a353aadeca0ada2443765b0a10b1ce9910709"}
Oct 29 23:00:03.248007 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.247701    4860 scope.go:122] "RemoveContainer" containerID="cfc58b1362ff99feacc8c7d4c66a353aadeca0ada2443765b0a10b1ce9910709"
Oct 29 23:00:03.249924 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.249897    4860 generic.go:356] "Generic (PLEG): container finished" podID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" containerID="56c0a9aecb99c44017a2f0225353929bccdba1ec0b9d178d4455b039fe52ee96" exitCode=60
Oct 29 23:00:03.250086 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.249971    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4" event={"ID":"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d","Type":"ContainerDied","Data":"56c0a9aecb99c44017a2f0225353929bccdba1ec0b9d178d4455b039fe52ee96"}
Oct 29 23:00:03.250204 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.250189    4860 scope.go:122] "RemoveContainer" containerID="56c0a9aecb99c44017a2f0225353929bccdba1ec0b9d178d4455b039fe52ee96"
Oct 29 23:00:03.253947 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.253920    4860 generic.go:356] "Generic (PLEG): container finished" podID="4e276a4c-e944-495d-8d4e-f80ebd436f05" containerID="6b89b07c8819c748bcd652c3505a518d766a9fc968c2a7f1514e5f6209459148" exitCode=60
Oct 29 23:00:03.254067 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.254046    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds" event={"ID":"4e276a4c-e944-495d-8d4e-f80ebd436f05","Type":"ContainerDied","Data":"6b89b07c8819c748bcd652c3505a518d766a9fc968c2a7f1514e5f6209459148"}
Oct 29 23:00:03.255224 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.254215    4860 scope.go:122] "RemoveContainer" containerID="6b89b07c8819c748bcd652c3505a518d766a9fc968c2a7f1514e5f6209459148"
Oct 29 23:00:03.257038 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.257008    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6" event={"ID":"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76","Type":"ContainerStarted","Data":"794341f16cd4e6388cec59266dbf2a6b9ab5af8b3abe6d4e668911df2e682f85"}
Oct 29 23:00:03.259915 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.259611    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks" event={"ID":"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c","Type":"ContainerStarted","Data":"70a3f37fd2c09da2622148d33b28cd08598051492a3c9c2d547b49750bef913d"}
Oct 29 23:00:03.263911 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.261974    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n" event={"ID":"7c7a942f-a12b-42b1-8b11-b6e2c5103798","Type":"ContainerStarted","Data":"c5e50cbde4bc16021ee6b6b39472b202af775f1fff2b7baccc5b9c64fbdc06c4"}
Oct 29 23:00:03.264030 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.263906    4860 generic.go:356] "Generic (PLEG): container finished" podID="521dba0b-944b-4ee0-8e00-60fa687c94bd" containerID="002e98022f3b0f202ddfb101d4b9e8886c3ba6931b0bdd5e4a2e1df21fdc21d4" exitCode=60
Oct 29 23:00:03.264030 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.264016    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl" event={"ID":"521dba0b-944b-4ee0-8e00-60fa687c94bd","Type":"ContainerDied","Data":"002e98022f3b0f202ddfb101d4b9e8886c3ba6931b0bdd5e4a2e1df21fdc21d4"}
Oct 29 23:00:03.264293 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.264276    4860 scope.go:122] "RemoveContainer" containerID="002e98022f3b0f202ddfb101d4b9e8886c3ba6931b0bdd5e4a2e1df21fdc21d4"
Oct 29 23:00:03.265673 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.265651    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5" event={"ID":"9f4a5ad2-3f37-4854-975d-355793e94b71","Type":"ContainerStarted","Data":"c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe"}
Oct 29 23:00:03.265737 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.265695    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5" event={"ID":"9f4a5ad2-3f37-4854-975d-355793e94b71","Type":"ContainerStarted","Data":"3b768c4fde3060505119fbaf92e9da7b3256ba71eb1a2ec9496c47ea1527eb0a"}
Oct 29 23:00:03.265875 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.265863    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"
Oct 29 23:00:03.327897 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.327839    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks" podStartSLOduration=3.327814108 podStartE2EDuration="3.327814108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.321487489 +0000 UTC m=+2745.346263622" watchObservedRunningTime="2026-10-29 23:00:03.327814108 +0000 UTC m=+2745.352590241"
Oct 29 23:00:03.409960 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.409902    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd" podStartSLOduration=3.4098846 podStartE2EDuration="3.4098846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.407178015 +0000 UTC m=+2745.431954150" watchObservedRunningTime="2026-10-29 23:00:03.4098846 +0000 UTC m=+2745.434660738"
Oct 29 23:00:03.451999 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.451888    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n" podStartSLOduration=3.451700627 podStartE2EDuration="3.451700627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.449641688 +0000 UTC m=+2745.474417842" watchObservedRunningTime="2026-10-29 23:00:03.451700627 +0000 UTC m=+2745.476476762"
Oct 29 23:00:03.534100 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.534046    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5" podStartSLOduration=2.5340284889999998 podStartE2EDuration="2.534028489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.533605383 +0000 UTC m=+2745.558381519" watchObservedRunningTime="2026-10-29 23:00:03.534028489 +0000 UTC m=+2745.558804622"
Oct 29 23:00:03.620600 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.620547    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6" podStartSLOduration=3.620528853 podStartE2EDuration="3.620528853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.613129844 +0000 UTC m=+2745.637906002" watchObservedRunningTime="2026-10-29 23:00:03.620528853 +0000 UTC m=+2745.645304985"
Oct 29 23:00:03.796383 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:03.796289    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9" podStartSLOduration=2.7962707509999998 podStartE2EDuration="2.796270751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.788504112 +0000 UTC m=+2745.813280244" watchObservedRunningTime="2026-10-29 23:00:03.796270751 +0000 UTC m=+2745.821046885"
Oct 29 23:00:04.282002 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.281924    4860 generic.go:356] "Generic (PLEG): container finished" podID="32dbf81d-0294-4922-b785-1a0b3950017a" containerID="4b06c41e2458d0c3d8c85166a9c8badf7bdedcc0671117dacabe3104c01808b0" exitCode=60
Oct 29 23:00:04.282511 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.281998    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd" event={"ID":"32dbf81d-0294-4922-b785-1a0b3950017a","Type":"ContainerDied","Data":"4b06c41e2458d0c3d8c85166a9c8badf7bdedcc0671117dacabe3104c01808b0"}
Oct 29 23:00:04.282511 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.282309    4860 scope.go:122] "RemoveContainer" containerID="4b06c41e2458d0c3d8c85166a9c8badf7bdedcc0671117dacabe3104c01808b0"
Oct 29 23:00:04.284405 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.284383    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk" event={"ID":"2247bdec-f506-4144-a05a-884528c5c132","Type":"ContainerStarted","Data":"b01db0d15e20cfc43f0da7809a68d4b85b5cb6ab35a7c4e006ab8a8a997b0f0b"}
Oct 29 23:00:04.286897 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.286870    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4" event={"ID":"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d","Type":"ContainerStarted","Data":"152fe06aa13b47948b2c08000df8be4cf933e068a37f8201accfd095d65f7a38"}
Oct 29 23:00:04.289086 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.289056    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds" event={"ID":"4e276a4c-e944-495d-8d4e-f80ebd436f05","Type":"ContainerStarted","Data":"a57ca4980909e1ae4971c6d5cf3ad2f188f39dbf535a2c52eadc1ad0340f06c1"}
Oct 29 23:00:04.290942 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.290919    4860 generic.go:356] "Generic (PLEG): container finished" podID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" containerID="794341f16cd4e6388cec59266dbf2a6b9ab5af8b3abe6d4e668911df2e682f85" exitCode=60
Oct 29 23:00:04.291051 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.290998    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6" event={"ID":"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76","Type":"ContainerDied","Data":"794341f16cd4e6388cec59266dbf2a6b9ab5af8b3abe6d4e668911df2e682f85"}
Oct 29 23:00:04.291286 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.291267    4860 scope.go:122] "RemoveContainer" containerID="794341f16cd4e6388cec59266dbf2a6b9ab5af8b3abe6d4e668911df2e682f85"
Oct 29 23:00:04.292855 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.292835    4860 generic.go:356] "Generic (PLEG): container finished" podID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" containerID="70a3f37fd2c09da2622148d33b28cd08598051492a3c9c2d547b49750bef913d" exitCode=60
Oct 29 23:00:04.292985 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.292926    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks" event={"ID":"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c","Type":"ContainerDied","Data":"70a3f37fd2c09da2622148d33b28cd08598051492a3c9c2d547b49750bef913d"}
Oct 29 23:00:04.293111 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.293096    4860 scope.go:122] "RemoveContainer" containerID="70a3f37fd2c09da2622148d33b28cd08598051492a3c9c2d547b49750bef913d"
Oct 29 23:00:04.294652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.294630    4860 generic.go:356] "Generic (PLEG): container finished" podID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" containerID="c5e50cbde4bc16021ee6b6b39472b202af775f1fff2b7baccc5b9c64fbdc06c4" exitCode=60
Oct 29 23:00:04.294750 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.294701    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n" event={"ID":"7c7a942f-a12b-42b1-8b11-b6e2c5103798","Type":"ContainerDied","Data":"c5e50cbde4bc16021ee6b6b39472b202af775f1fff2b7baccc5b9c64fbdc06c4"}
Oct 29 23:00:04.294945 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.294927    4860 scope.go:122] "RemoveContainer" containerID="c5e50cbde4bc16021ee6b6b39472b202af775f1fff2b7baccc5b9c64fbdc06c4"
Oct 29 23:00:04.297471 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.297439    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl" event={"ID":"521dba0b-944b-4ee0-8e00-60fa687c94bd","Type":"ContainerStarted","Data":"6fab43b6b66ecaa3bedec595b768129a51ba60d67ff3cfbb412bda0cc37c692c"}
Oct 29 23:00:04.303243 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.303220    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k" event={"ID":"b151b922-7435-4d4e-a403-3c911de5d8f6","Type":"ContainerStarted","Data":"97202d57693f929a0ab4035108776d2a9040d8d6036e9b1fb334ec2cc852e751"}
Oct 29 23:00:04.306008 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.305989    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb" event={"ID":"0b239203-2773-4ff8-bc28-604940a34500","Type":"ContainerStarted","Data":"42a43916634bee886ba292257bb235b1f14b6908239a0adb808bd6928569dfe3"}
Oct 29 23:00:04.308640 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.308617    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86" event={"ID":"2f461145-f50b-4ebd-b9fa-25edb7d2af1a","Type":"ContainerStarted","Data":"054f657249b963eb4a424f5fae35a2edbac59211917a06f725decef74ded6db3"}
Oct 29 23:00:04.316385 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.316360    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt" event={"ID":"cb80ca82-acc6-4bdf-b76a-f39eb983f49a","Type":"ContainerStarted","Data":"785eac769fae9a384aa49ac78f663fd72a15e1bcb07d622f4ffc1604d5ebf6f6"}
Oct 29 23:00:04.409581 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.409523    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl" podStartSLOduration=4.409506298 podStartE2EDuration="4.409506298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:04.407882005 +0000 UTC m=+2746.432658138" watchObservedRunningTime="2026-10-29 23:00:04.409506298 +0000 UTC m=+2746.434282430"
Oct 29 23:00:04.454191 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.454144    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb" podStartSLOduration=4.454131432 podStartE2EDuration="4.454131432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:04.452397332 +0000 UTC m=+2746.477173494" watchObservedRunningTime="2026-10-29 23:00:04.454131432 +0000 UTC m=+2746.478907587"
Oct 29 23:00:04.538482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.538381    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds" podStartSLOduration=4.538367125 podStartE2EDuration="4.538367125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:04.534776404 +0000 UTC m=+2746.559552553" watchObservedRunningTime="2026-10-29 23:00:04.538367125 +0000 UTC m=+2746.563143258"
Oct 29 23:00:04.672743 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.672710    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:04.731492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:04.731426    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k" podStartSLOduration=4.731403116 podStartE2EDuration="4.731403116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:04.721099079 +0000 UTC m=+2746.745875210" watchObservedRunningTime="2026-10-29 23:00:04.731403116 +0000 UTC m=+2746.756179283"
Oct 29 23:00:04.920363 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:04.920313    4860 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2247bdec_f506_4144_a05a_884528c5c132.slice/crio-conmon-b01db0d15e20cfc43f0da7809a68d4b85b5cb6ab35a7c4e006ab8a8a997b0f0b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521dba0b_944b_4ee0_8e00_60fa687c94bd.slice/crio-conmon-6fab43b6b66ecaa3bedec595b768129a51ba60d67ff3cfbb412bda0cc37c692c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2247bdec_f506_4144_a05a_884528c5c132.slice/crio-b01db0d15e20cfc43f0da7809a68d4b85b5cb6ab35a7c4e006ab8a8a997b0f0b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb151b922_7435_4d4e_a403_3c911de5d8f6.slice/crio-conmon-97202d57693f929a0ab4035108776d2a9040d8d6036e9b1fb334ec2cc852e751.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 23:00:05.329282 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.329207    4860 generic.go:356] "Generic (PLEG): container finished" podID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" containerID="152fe06aa13b47948b2c08000df8be4cf933e068a37f8201accfd095d65f7a38" exitCode=60
Oct 29 23:00:05.329725 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.329288    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4" event={"ID":"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d","Type":"ContainerDied","Data":"152fe06aa13b47948b2c08000df8be4cf933e068a37f8201accfd095d65f7a38"}
Oct 29 23:00:05.329725 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.329408    4860 scope.go:122] "RemoveContainer" containerID="56c0a9aecb99c44017a2f0225353929bccdba1ec0b9d178d4455b039fe52ee96"
Oct 29 23:00:05.329725 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.329698    4860 scope.go:122] "RemoveContainer" containerID="152fe06aa13b47948b2c08000df8be4cf933e068a37f8201accfd095d65f7a38"
Oct 29 23:00:05.330164 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:05.330136    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4_odh-ai-gateway-infra(bf8314b5-c567-4ebd-8a0b-d55cf6b3047d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d"
Oct 29 23:00:05.332393 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.332370    4860 generic.go:356] "Generic (PLEG): container finished" podID="4e276a4c-e944-495d-8d4e-f80ebd436f05" containerID="a57ca4980909e1ae4971c6d5cf3ad2f188f39dbf535a2c52eadc1ad0340f06c1" exitCode=60
Oct 29 23:00:05.332492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.332449    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds" event={"ID":"4e276a4c-e944-495d-8d4e-f80ebd436f05","Type":"ContainerDied","Data":"a57ca4980909e1ae4971c6d5cf3ad2f188f39dbf535a2c52eadc1ad0340f06c1"}
Oct 29 23:00:05.332723 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.332707    4860 scope.go:122] "RemoveContainer" containerID="a57ca4980909e1ae4971c6d5cf3ad2f188f39dbf535a2c52eadc1ad0340f06c1"
Oct 29 23:00:05.333074 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:05.333051    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds_odh-ai-gateway-infra(4e276a4c-e944-495d-8d4e-f80ebd436f05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05"
Oct 29 23:00:05.334558 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.334533    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6" event={"ID":"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76","Type":"ContainerStarted","Data":"3549fb7a48e4a5b4d0f3420e33cbb8958737b99252afb4629d5abe56895e875b"}
Oct 29 23:00:05.336752 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.336726    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks" event={"ID":"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c","Type":"ContainerStarted","Data":"3c4e44c7ebc9235e965ab382fb1b08f3d54fcb80b37f3d892feb2d642ff2d6d3"}
Oct 29 23:00:05.339268 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.339246    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n" event={"ID":"7c7a942f-a12b-42b1-8b11-b6e2c5103798","Type":"ContainerStarted","Data":"264fc49da7b2a82795f866e48cd9a0c91f1584c85c229db749acef5fb4538992"}
Oct 29 23:00:05.341364 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.341341    4860 generic.go:356] "Generic (PLEG): container finished" podID="521dba0b-944b-4ee0-8e00-60fa687c94bd" containerID="6fab43b6b66ecaa3bedec595b768129a51ba60d67ff3cfbb412bda0cc37c692c" exitCode=60
Oct 29 23:00:05.341451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.341420    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl" event={"ID":"521dba0b-944b-4ee0-8e00-60fa687c94bd","Type":"ContainerDied","Data":"6fab43b6b66ecaa3bedec595b768129a51ba60d67ff3cfbb412bda0cc37c692c"}
Oct 29 23:00:05.341593 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.341577    4860 scope.go:122] "RemoveContainer" containerID="6fab43b6b66ecaa3bedec595b768129a51ba60d67ff3cfbb412bda0cc37c692c"
Oct 29 23:00:05.341862 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:05.341836    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl_odh-ai-gateway-infra(521dba0b-944b-4ee0-8e00-60fa687c94bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd"
Oct 29 23:00:05.343606 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.343583    4860 generic.go:356] "Generic (PLEG): container finished" podID="b151b922-7435-4d4e-a403-3c911de5d8f6" containerID="97202d57693f929a0ab4035108776d2a9040d8d6036e9b1fb334ec2cc852e751" exitCode=60
Oct 29 23:00:05.343691 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.343663    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k" event={"ID":"b151b922-7435-4d4e-a403-3c911de5d8f6","Type":"ContainerDied","Data":"97202d57693f929a0ab4035108776d2a9040d8d6036e9b1fb334ec2cc852e751"}
Oct 29 23:00:05.343853 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.343839    4860 scope.go:122] "RemoveContainer" containerID="97202d57693f929a0ab4035108776d2a9040d8d6036e9b1fb334ec2cc852e751"
Oct 29 23:00:05.344105 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:05.344086    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k_odh-ai-gateway-infra(b151b922-7435-4d4e-a403-3c911de5d8f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6"
Oct 29 23:00:05.345649 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.345626    4860 generic.go:356] "Generic (PLEG): container finished" podID="0b239203-2773-4ff8-bc28-604940a34500" containerID="42a43916634bee886ba292257bb235b1f14b6908239a0adb808bd6928569dfe3" exitCode=60
Oct 29 23:00:05.345755 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.345694    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb" event={"ID":"0b239203-2773-4ff8-bc28-604940a34500","Type":"ContainerDied","Data":"42a43916634bee886ba292257bb235b1f14b6908239a0adb808bd6928569dfe3"}
Oct 29 23:00:05.345937 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.345923    4860 scope.go:122] "RemoveContainer" containerID="42a43916634bee886ba292257bb235b1f14b6908239a0adb808bd6928569dfe3"
Oct 29 23:00:05.346199 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:05.346180    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb_odh-ai-gateway-infra(0b239203-2773-4ff8-bc28-604940a34500)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb" podUID="0b239203-2773-4ff8-bc28-604940a34500"
Oct 29 23:00:05.346710 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.346691    4860 scope.go:122] "RemoveContainer" containerID="6b89b07c8819c748bcd652c3505a518d766a9fc968c2a7f1514e5f6209459148"
Oct 29 23:00:05.347712 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.347696    4860 generic.go:356] "Generic (PLEG): container finished" podID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" containerID="054f657249b963eb4a424f5fae35a2edbac59211917a06f725decef74ded6db3" exitCode=60
Oct 29 23:00:05.347803 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.347791    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86" event={"ID":"2f461145-f50b-4ebd-b9fa-25edb7d2af1a","Type":"ContainerDied","Data":"054f657249b963eb4a424f5fae35a2edbac59211917a06f725decef74ded6db3"}
Oct 29 23:00:05.348073 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.348059    4860 scope.go:122] "RemoveContainer" containerID="054f657249b963eb4a424f5fae35a2edbac59211917a06f725decef74ded6db3"
Oct 29 23:00:05.348316 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:05.348294    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86_odh-ai-gateway-infra(2f461145-f50b-4ebd-b9fa-25edb7d2af1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a"
Oct 29 23:00:05.349780 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.349761    4860 generic.go:356] "Generic (PLEG): container finished" podID="cb80ca82-acc6-4bdf-b76a-f39eb983f49a" containerID="785eac769fae9a384aa49ac78f663fd72a15e1bcb07d622f4ffc1604d5ebf6f6" exitCode=60
Oct 29 23:00:05.349869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.349837    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt" event={"ID":"cb80ca82-acc6-4bdf-b76a-f39eb983f49a","Type":"ContainerDied","Data":"785eac769fae9a384aa49ac78f663fd72a15e1bcb07d622f4ffc1604d5ebf6f6"}
Oct 29 23:00:05.350040 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.350022    4860 scope.go:122] "RemoveContainer" containerID="785eac769fae9a384aa49ac78f663fd72a15e1bcb07d622f4ffc1604d5ebf6f6"
Oct 29 23:00:05.350278 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:05.350256    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt_odh-ai-gateway-infra(cb80ca82-acc6-4bdf-b76a-f39eb983f49a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt" podUID="cb80ca82-acc6-4bdf-b76a-f39eb983f49a"
Oct 29 23:00:05.351767 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.351745    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd" event={"ID":"32dbf81d-0294-4922-b785-1a0b3950017a","Type":"ContainerStarted","Data":"44c90ab611df18c2b7a2787656aad9de8896a3f5ab3b8879f7c1d6667b03375c"}
Oct 29 23:00:05.353791 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.353772    4860 generic.go:356] "Generic (PLEG): container finished" podID="2247bdec-f506-4144-a05a-884528c5c132" containerID="b01db0d15e20cfc43f0da7809a68d4b85b5cb6ab35a7c4e006ab8a8a997b0f0b" exitCode=60
Oct 29 23:00:05.353893 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.353830    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk" event={"ID":"2247bdec-f506-4144-a05a-884528c5c132","Type":"ContainerDied","Data":"b01db0d15e20cfc43f0da7809a68d4b85b5cb6ab35a7c4e006ab8a8a997b0f0b"}
Oct 29 23:00:05.354011 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.353997    4860 scope.go:122] "RemoveContainer" containerID="b01db0d15e20cfc43f0da7809a68d4b85b5cb6ab35a7c4e006ab8a8a997b0f0b"
Oct 29 23:00:05.354231 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:05.354214    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk_odh-ai-gateway-infra(2247bdec-f506-4144-a05a-884528c5c132)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk" podUID="2247bdec-f506-4144-a05a-884528c5c132"
Oct 29 23:00:05.361900 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.361882    4860 scope.go:122] "RemoveContainer" containerID="002e98022f3b0f202ddfb101d4b9e8886c3ba6931b0bdd5e4a2e1df21fdc21d4"
Oct 29 23:00:05.380016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.379999    4860 scope.go:122] "RemoveContainer" containerID="22c07c0178a9c09fc402bb5338b6c1200787132752c20bfc6c21f30cb4fba202"
Oct 29 23:00:05.394880 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.394854    4860 scope.go:122] "RemoveContainer" containerID="c42bef62738652eb5ab1f534aa4368c9f21b868e50cf0fa94da2305ebd3ff976"
Oct 29 23:00:05.409887 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.409870    4860 scope.go:122] "RemoveContainer" containerID="a27588bf8ed6b63ab08a8857aac9cc0a16c4756f3a97aa7974085eda3c786058"
Oct 29 23:00:05.425087 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.425068    4860 scope.go:122] "RemoveContainer" containerID="81231caf2b89eb2ce7e623a5dfca7115eb780d4bcbfb74ec5a7d6efb506e7f95"
Oct 29 23:00:05.445093 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:05.445076    4860 scope.go:122] "RemoveContainer" containerID="cfc58b1362ff99feacc8c7d4c66a353aadeca0ada2443765b0a10b1ce9910709"
Oct 29 23:00:06.365754 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.365656    4860 scope.go:122] "RemoveContainer" containerID="97202d57693f929a0ab4035108776d2a9040d8d6036e9b1fb334ec2cc852e751"
Oct 29 23:00:06.366201 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.366129    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k_odh-ai-gateway-infra(b151b922-7435-4d4e-a403-3c911de5d8f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6"
Oct 29 23:00:06.367545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.367522    4860 scope.go:122] "RemoveContainer" containerID="42a43916634bee886ba292257bb235b1f14b6908239a0adb808bd6928569dfe3"
Oct 29 23:00:06.367916 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.367890    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb_odh-ai-gateway-infra(0b239203-2773-4ff8-bc28-604940a34500)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb" podUID="0b239203-2773-4ff8-bc28-604940a34500"
Oct 29 23:00:06.369207 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.369185    4860 scope.go:122] "RemoveContainer" containerID="054f657249b963eb4a424f5fae35a2edbac59211917a06f725decef74ded6db3"
Oct 29 23:00:06.369592 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.369569    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86_odh-ai-gateway-infra(2f461145-f50b-4ebd-b9fa-25edb7d2af1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a"
Oct 29 23:00:06.370860 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.370838    4860 scope.go:122] "RemoveContainer" containerID="785eac769fae9a384aa49ac78f663fd72a15e1bcb07d622f4ffc1604d5ebf6f6"
Oct 29 23:00:06.371222 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.371194    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt_odh-ai-gateway-infra(cb80ca82-acc6-4bdf-b76a-f39eb983f49a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt" podUID="cb80ca82-acc6-4bdf-b76a-f39eb983f49a"
Oct 29 23:00:06.372585 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.372564    4860 generic.go:356] "Generic (PLEG): container finished" podID="32dbf81d-0294-4922-b785-1a0b3950017a" containerID="44c90ab611df18c2b7a2787656aad9de8896a3f5ab3b8879f7c1d6667b03375c" exitCode=60
Oct 29 23:00:06.372683 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.372650    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd" event={"ID":"32dbf81d-0294-4922-b785-1a0b3950017a","Type":"ContainerDied","Data":"44c90ab611df18c2b7a2787656aad9de8896a3f5ab3b8879f7c1d6667b03375c"}
Oct 29 23:00:06.372768 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.372714    4860 scope.go:122] "RemoveContainer" containerID="4b06c41e2458d0c3d8c85166a9c8badf7bdedcc0671117dacabe3104c01808b0"
Oct 29 23:00:06.372860 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.372845    4860 scope.go:122] "RemoveContainer" containerID="44c90ab611df18c2b7a2787656aad9de8896a3f5ab3b8879f7c1d6667b03375c"
Oct 29 23:00:06.373171 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.373148    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd_odh-ai-gateway-infra(32dbf81d-0294-4922-b785-1a0b3950017a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd" podUID="32dbf81d-0294-4922-b785-1a0b3950017a"
Oct 29 23:00:06.374692 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.374674    4860 scope.go:122] "RemoveContainer" containerID="b01db0d15e20cfc43f0da7809a68d4b85b5cb6ab35a7c4e006ab8a8a997b0f0b"
Oct 29 23:00:06.374957 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.374935    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk_odh-ai-gateway-infra(2247bdec-f506-4144-a05a-884528c5c132)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk" podUID="2247bdec-f506-4144-a05a-884528c5c132"
Oct 29 23:00:06.376353 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.376320    4860 scope.go:122] "RemoveContainer" containerID="152fe06aa13b47948b2c08000df8be4cf933e068a37f8201accfd095d65f7a38"
Oct 29 23:00:06.376617 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.376594    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4_odh-ai-gateway-infra(bf8314b5-c567-4ebd-8a0b-d55cf6b3047d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d"
Oct 29 23:00:06.377957 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.377939    4860 scope.go:122] "RemoveContainer" containerID="a57ca4980909e1ae4971c6d5cf3ad2f188f39dbf535a2c52eadc1ad0340f06c1"
Oct 29 23:00:06.378202 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.378180    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds_odh-ai-gateway-infra(4e276a4c-e944-495d-8d4e-f80ebd436f05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05"
Oct 29 23:00:06.379884 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.379863    4860 generic.go:356] "Generic (PLEG): container finished" podID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" containerID="3549fb7a48e4a5b4d0f3420e33cbb8958737b99252afb4629d5abe56895e875b" exitCode=60
Oct 29 23:00:06.379977 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.379952    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6" event={"ID":"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76","Type":"ContainerDied","Data":"3549fb7a48e4a5b4d0f3420e33cbb8958737b99252afb4629d5abe56895e875b"}
Oct 29 23:00:06.380165 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.380148    4860 scope.go:122] "RemoveContainer" containerID="3549fb7a48e4a5b4d0f3420e33cbb8958737b99252afb4629d5abe56895e875b"
Oct 29 23:00:06.380456 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.380430    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6_odh-ai-gateway-infra(e27bd901-d6a1-48c3-91fe-b9c12b3d7b76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76"
Oct 29 23:00:06.381989 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.381967    4860 generic.go:356] "Generic (PLEG): container finished" podID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" containerID="3c4e44c7ebc9235e965ab382fb1b08f3d54fcb80b37f3d892feb2d642ff2d6d3" exitCode=60
Oct 29 23:00:06.382084 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.382050    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks" event={"ID":"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c","Type":"ContainerDied","Data":"3c4e44c7ebc9235e965ab382fb1b08f3d54fcb80b37f3d892feb2d642ff2d6d3"}
Oct 29 23:00:06.382269 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.382252    4860 scope.go:122] "RemoveContainer" containerID="3c4e44c7ebc9235e965ab382fb1b08f3d54fcb80b37f3d892feb2d642ff2d6d3"
Oct 29 23:00:06.382587 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.382564    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks_odh-ai-gateway-infra(4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c"
Oct 29 23:00:06.383986 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.383963    4860 generic.go:356] "Generic (PLEG): container finished" podID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" containerID="264fc49da7b2a82795f866e48cd9a0c91f1584c85c229db749acef5fb4538992" exitCode=60
Oct 29 23:00:06.384084 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.383987    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n" event={"ID":"7c7a942f-a12b-42b1-8b11-b6e2c5103798","Type":"ContainerDied","Data":"264fc49da7b2a82795f866e48cd9a0c91f1584c85c229db749acef5fb4538992"}
Oct 29 23:00:06.384233 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.384217    4860 scope.go:122] "RemoveContainer" containerID="264fc49da7b2a82795f866e48cd9a0c91f1584c85c229db749acef5fb4538992"
Oct 29 23:00:06.384466 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.384446    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n_odh-ai-gateway-infra(7c7a942f-a12b-42b1-8b11-b6e2c5103798)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798"
Oct 29 23:00:06.385995 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.385973    4860 scope.go:122] "RemoveContainer" containerID="6fab43b6b66ecaa3bedec595b768129a51ba60d67ff3cfbb412bda0cc37c692c"
Oct 29 23:00:06.386237 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.386215    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl_odh-ai-gateway-infra(521dba0b-944b-4ee0-8e00-60fa687c94bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd"
Oct 29 23:00:06.389759 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.389731    4860 scope.go:122] "RemoveContainer" containerID="794341f16cd4e6388cec59266dbf2a6b9ab5af8b3abe6d4e668911df2e682f85"
Oct 29 23:00:06.404262 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.404245    4860 scope.go:122] "RemoveContainer" containerID="70a3f37fd2c09da2622148d33b28cd08598051492a3c9c2d547b49750bef913d"
Oct 29 23:00:06.419480 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.419461    4860 scope.go:122] "RemoveContainer" containerID="c5e50cbde4bc16021ee6b6b39472b202af775f1fff2b7baccc5b9c64fbdc06c4"
Oct 29 23:00:06.695382 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.695351    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"]
Oct 29 23:00:06.701800 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.701779    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:06.719157 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.719129    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-75b68d-serving-cert\""
Oct 29 23:00:06.737337 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.737292    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"]
Oct 29 23:00:06.786889 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.786829    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:06.787451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.786977    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-metrics-certs\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:06.787451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.787125    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-624rl\" (UniqueName: \"kubernetes.io/projected/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-kube-api-access-624rl\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:06.787812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.787775    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:06.805168 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.805077    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:06.918348 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.905509    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-624rl\" (UniqueName: \"kubernetes.io/projected/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-kube-api-access-624rl\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:06.918348 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.905644    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:06.918348 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.905894    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:06.918348 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.905970    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-metrics-certs\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:06.918761 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.918582    4860 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-serving-cert: secret "maas-api-e2e-ipp-b-75b68d-serving-cert" not found
Oct 29 23:00:06.918761 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:06.918667    4860 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-maas-api-tls podName:2a4ac286-93cf-4aaf-a3da-80c5f28417bb nodeName:}" failed. No retries permitted until 2026-10-29 23:00:07.418638732 +0000 UTC m=+2749.443414856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-maas-api-tls") pod "maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv" (UID: "2a4ac286-93cf-4aaf-a3da-80c5f28417bb") : secret "maas-api-e2e-ipp-b-75b68d-serving-cert" not found
Oct 29 23:00:06.933812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.919992    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:06.940424 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.934630    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:06.940424 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.938653    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-metrics-certs\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:06.950922 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:06.947926    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-624rl\" (UniqueName: \"kubernetes.io/projected/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-kube-api-access-624rl\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:07.153809 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.153766    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:07.313912 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.313800    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:07.397452 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.397420    4860 scope.go:122] "RemoveContainer" containerID="44c90ab611df18c2b7a2787656aad9de8896a3f5ab3b8879f7c1d6667b03375c"
Oct 29 23:00:07.397907 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:07.397877    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd_odh-ai-gateway-infra(32dbf81d-0294-4922-b785-1a0b3950017a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd" podUID="32dbf81d-0294-4922-b785-1a0b3950017a"
Oct 29 23:00:07.399474 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.399453    4860 scope.go:122] "RemoveContainer" containerID="3549fb7a48e4a5b4d0f3420e33cbb8958737b99252afb4629d5abe56895e875b"
Oct 29 23:00:07.399825 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:07.399805    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6_odh-ai-gateway-infra(e27bd901-d6a1-48c3-91fe-b9c12b3d7b76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76"
Oct 29 23:00:07.401284 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.401260    4860 scope.go:122] "RemoveContainer" containerID="3c4e44c7ebc9235e965ab382fb1b08f3d54fcb80b37f3d892feb2d642ff2d6d3"
Oct 29 23:00:07.401604 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:07.401578    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks_odh-ai-gateway-infra(4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c"
Oct 29 23:00:07.402927 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.402907    4860 scope.go:122] "RemoveContainer" containerID="264fc49da7b2a82795f866e48cd9a0c91f1584c85c229db749acef5fb4538992"
Oct 29 23:00:07.403197 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:07.403179    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n_odh-ai-gateway-infra(7c7a942f-a12b-42b1-8b11-b6e2c5103798)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798"
Oct 29 23:00:07.518354 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.518275    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:07.541864 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.541826    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:07.551662 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.551630    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"]
Oct 29 23:00:07.551908 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.551884    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv" podUID="bffcbfcb-f8c9-441f-a456-b55829cad058" containerName="istio-proxy" containerID="cri-o://5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58" gracePeriod=30
Oct 29 23:00:07.560777 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.560748    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"]
Oct 29 23:00:07.560980 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.560957    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm" podUID="453867f4-909e-4d42-8ac8-6ab6da4f0626" containerName="istio-proxy" containerID="cri-o://ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b" gracePeriod=30
Oct 29 23:00:07.567049 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.566967    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"]
Oct 29 23:00:07.567187 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.567163    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b" podUID="7c7b50c2-573d-41c0-b372-b51fd1e29ba6" containerName="istio-proxy" containerID="cri-o://20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878" gracePeriod=30
Oct 29 23:00:07.572510 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.572445    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:07.636669 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:07.636631    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:08.375749 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:00:08.375713    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4ac286_93cf_4aaf_a3da_80c5f28417bb.slice/crio-11d9338fb554e66d9223d6831aad316bb83be371656ab953f5cf6939a5f28106 WatchSource:0}: Error finding container 11d9338fb554e66d9223d6831aad316bb83be371656ab953f5cf6939a5f28106: Status 404 returned error can't find the container with id 11d9338fb554e66d9223d6831aad316bb83be371656ab953f5cf6939a5f28106
Oct 29 23:00:08.378843 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:08.378817    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"]
Oct 29 23:00:08.414745 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:08.414717    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv" event={"ID":"2a4ac286-93cf-4aaf-a3da-80c5f28417bb","Type":"ContainerStarted","Data":"11d9338fb554e66d9223d6831aad316bb83be371656ab953f5cf6939a5f28106"}
Oct 29 23:00:08.855433 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:08.855402    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:09.426269 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:09.426229    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv" event={"ID":"2a4ac286-93cf-4aaf-a3da-80c5f28417bb","Type":"ContainerStarted","Data":"60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048"}
Oct 29 23:00:09.426804 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:09.426347    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:09.476279 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:09.476231    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv" podStartSLOduration=3.214390801 podStartE2EDuration="3.476218454s" totalImagesPullingTime="261.827653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:09.470375613 +0000 UTC m=+2751.495151762" watchObservedRunningTime="2026-10-29 23:00:09.476218454 +0000 UTC m=+2751.500994586"
Oct 29 23:00:10.450552 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:10.450518    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"]
Oct 29 23:00:10.465540 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:10.465509    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:10.729085 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:10.728996    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"
Oct 29 23:00:10.819345 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:10.815987    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-maas-api-service-ca\") pod \"cb80ca82-acc6-4bdf-b76a-f39eb983f49a\" (UID: \"cb80ca82-acc6-4bdf-b76a-f39eb983f49a\") "
Oct 29 23:00:10.819345 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:10.816094    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-kube-api-access-lzw48\" (UniqueName: \"kubernetes.io/projected/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-kube-api-access-lzw48\") pod \"cb80ca82-acc6-4bdf-b76a-f39eb983f49a\" (UID: \"cb80ca82-acc6-4bdf-b76a-f39eb983f49a\") "
Oct 29 23:00:10.829142 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:10.828745    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-maas-api-service-ca" pod "cb80ca82-acc6-4bdf-b76a-f39eb983f49a" (UID: "cb80ca82-acc6-4bdf-b76a-f39eb983f49a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:10.875378 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:10.864309    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-kube-api-access-lzw48" pod "cb80ca82-acc6-4bdf-b76a-f39eb983f49a" (UID: "cb80ca82-acc6-4bdf-b76a-f39eb983f49a"). InnerVolumeSpecName "kube-api-access-lzw48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:10.933521 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:10.931198    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:10.933521 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:10.931238    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzw48\" (UniqueName: \"kubernetes.io/projected/cb80ca82-acc6-4bdf-b76a-f39eb983f49a-kube-api-access-lzw48\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:10.969619 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:10.969446    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:11.105642 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.102350    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:11.236858 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.236729    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:11.277853 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.277826    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"
Oct 29 23:00:11.362164 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.362126    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e951ea-8eb5-4a3a-aabf-dde45406c921-kube-api-access-6xg48\" (UniqueName: \"kubernetes.io/projected/f5e951ea-8eb5-4a3a-aabf-dde45406c921-kube-api-access-6xg48\") pod \"f5e951ea-8eb5-4a3a-aabf-dde45406c921\" (UID: \"f5e951ea-8eb5-4a3a-aabf-dde45406c921\") "
Oct 29 23:00:11.362384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.362280    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e951ea-8eb5-4a3a-aabf-dde45406c921-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e951ea-8eb5-4a3a-aabf-dde45406c921-plugins-config-volume\") pod \"f5e951ea-8eb5-4a3a-aabf-dde45406c921\" (UID: \"f5e951ea-8eb5-4a3a-aabf-dde45406c921\") "
Oct 29 23:00:11.362847 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.362811    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e951ea-8eb5-4a3a-aabf-dde45406c921-plugins-config-volume" pod "f5e951ea-8eb5-4a3a-aabf-dde45406c921" (UID: "f5e951ea-8eb5-4a3a-aabf-dde45406c921"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:11.363404 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.363378    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e951ea-8eb5-4a3a-aabf-dde45406c921-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:11.365702 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.365669    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e951ea-8eb5-4a3a-aabf-dde45406c921-kube-api-access-6xg48" pod "f5e951ea-8eb5-4a3a-aabf-dde45406c921" (UID: "f5e951ea-8eb5-4a3a-aabf-dde45406c921"). InnerVolumeSpecName "kube-api-access-6xg48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:11.409827 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.409802    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"
Oct 29 23:00:11.450584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.450548    4860 generic.go:356] "Generic (PLEG): container finished" podID="f5e951ea-8eb5-4a3a-aabf-dde45406c921" containerID="c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0" exitCode=0
Oct 29 23:00:11.451006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.450598    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"
Oct 29 23:00:11.451006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.450634    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx" event={"ID":"f5e951ea-8eb5-4a3a-aabf-dde45406c921","Type":"ContainerDied","Data":"c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0"}
Oct 29 23:00:11.451006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.450694    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx" event={"ID":"f5e951ea-8eb5-4a3a-aabf-dde45406c921","Type":"ContainerDied","Data":"e40028696e1e6de26bc77ad1e7ab727c4c3e2250d5ffb2df5045fb379100881b"}
Oct 29 23:00:11.451006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.450730    4860 scope.go:122] "RemoveContainer" containerID="c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0"
Oct 29 23:00:11.453482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.453244    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"
Oct 29 23:00:11.453482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.453354    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt" event={"ID":"cb80ca82-acc6-4bdf-b76a-f39eb983f49a","Type":"ContainerDied","Data":"e01d8db7e89ace85ed3a736d46a305b46993aaee4b10dad8aaaeaf7fb7d52bac"}
Oct 29 23:00:11.460148 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.460082    4860 generic.go:356] "Generic (PLEG): container finished" podID="4aaed535-f4bd-430d-8528-af2feb5a8ca4" containerID="9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc" exitCode=0
Oct 29 23:00:11.460148 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.460145    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw" event={"ID":"4aaed535-f4bd-430d-8528-af2feb5a8ca4","Type":"ContainerDied","Data":"9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc"}
Oct 29 23:00:11.460381 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.460155    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"
Oct 29 23:00:11.460381 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.460181    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw" event={"ID":"4aaed535-f4bd-430d-8528-af2feb5a8ca4","Type":"ContainerDied","Data":"9ba8aef88994da67701631c165c10366b293d92bca55d9079a7edf14cd2c4e1a"}
Oct 29 23:00:11.465235 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.465208    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xg48\" (UniqueName: \"kubernetes.io/projected/f5e951ea-8eb5-4a3a-aabf-dde45406c921-kube-api-access-6xg48\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:11.466604 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.466588    4860 scope.go:122] "RemoveContainer" containerID="c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0"
Oct 29 23:00:11.466918 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:11.466896    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0\": container with ID starting with c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0 not found: ID does not exist" containerID="c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0"
Oct 29 23:00:11.467016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.466928    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0"} err="failed to get container status \"c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0\": rpc error: code = NotFound desc = could not find container \"c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0\": container with ID starting with c2a723d626bf7ae4db59af6ed0d6d775225f1f718c31ddae22cb205b767b87f0 not found: ID does not exist"
Oct 29 23:00:11.467016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.466950    4860 scope.go:122] "RemoveContainer" containerID="785eac769fae9a384aa49ac78f663fd72a15e1bcb07d622f4ffc1604d5ebf6f6"
Oct 29 23:00:11.482786 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.482760    4860 scope.go:122] "RemoveContainer" containerID="9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc"
Oct 29 23:00:11.494433 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.494403    4860 scope.go:122] "RemoveContainer" containerID="9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc"
Oct 29 23:00:11.494728 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:11.494703    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc\": container with ID starting with 9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc not found: ID does not exist" containerID="9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc"
Oct 29 23:00:11.494810 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.494735    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc"} err="failed to get container status \"9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc\": rpc error: code = NotFound desc = could not find container \"9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc\": container with ID starting with 9e349f8c60851deb2b96398041b3c88798f0c8bbdaf0490802e5fed4dd31addc not found: ID does not exist"
Oct 29 23:00:11.557859 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.557837    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"]
Oct 29 23:00:11.563649 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.563625    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aaa3c6-29888580-dxqqt"]
Oct 29 23:00:11.566701 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.566678    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aaed535-f4bd-430d-8528-af2feb5a8ca4-kube-api-access-hnb86\" (UniqueName: \"kubernetes.io/projected/4aaed535-f4bd-430d-8528-af2feb5a8ca4-kube-api-access-hnb86\") pod \"4aaed535-f4bd-430d-8528-af2feb5a8ca4\" (UID: \"4aaed535-f4bd-430d-8528-af2feb5a8ca4\") "
Oct 29 23:00:11.566896 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.566879    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aaed535-f4bd-430d-8528-af2feb5a8ca4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaed535-f4bd-430d-8528-af2feb5a8ca4-plugins-config-volume\") pod \"4aaed535-f4bd-430d-8528-af2feb5a8ca4\" (UID: \"4aaed535-f4bd-430d-8528-af2feb5a8ca4\") "
Oct 29 23:00:11.567224 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.567202    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aaed535-f4bd-430d-8528-af2feb5a8ca4-plugins-config-volume" pod "4aaed535-f4bd-430d-8528-af2feb5a8ca4" (UID: "4aaed535-f4bd-430d-8528-af2feb5a8ca4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:11.567549 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.567533    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aaed535-f4bd-430d-8528-af2feb5a8ca4-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:11.568715 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.568697    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aaed535-f4bd-430d-8528-af2feb5a8ca4-kube-api-access-hnb86" pod "4aaed535-f4bd-430d-8528-af2feb5a8ca4" (UID: "4aaed535-f4bd-430d-8528-af2feb5a8ca4"). InnerVolumeSpecName "kube-api-access-hnb86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:11.595602 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.595575    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:11.596313 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.596291    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"]
Oct 29 23:00:11.601118 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.601098    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-784e7a-79d748c8bb-fbgzx"]
Oct 29 23:00:11.669468 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.669442    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnb86\" (UniqueName: \"kubernetes.io/projected/4aaed535-f4bd-430d-8528-af2feb5a8ca4-kube-api-access-hnb86\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:11.813857 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.813831    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"]
Oct 29 23:00:11.819212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:11.819161    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-784e7a-5bd4bc7d47-cddfw"]
Oct 29 23:00:12.117471 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:12.117382    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aaed535-f4bd-430d-8528-af2feb5a8ca4" path="/var/lib/kubelet/pods/4aaed535-f4bd-430d-8528-af2feb5a8ca4/volumes"
Oct 29 23:00:12.118121 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:12.118097    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb80ca82-acc6-4bdf-b76a-f39eb983f49a" path="/var/lib/kubelet/pods/cb80ca82-acc6-4bdf-b76a-f39eb983f49a/volumes"
Oct 29 23:00:12.118921 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:12.118899    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e951ea-8eb5-4a3a-aabf-dde45406c921" path="/var/lib/kubelet/pods/f5e951ea-8eb5-4a3a-aabf-dde45406c921/volumes"
Oct 29 23:00:12.914315 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:12.914284    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 23:00:12.918411 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:12.918388    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 23:00:12.987211 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:12.987187    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097245    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-socket\") pod \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097399    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-token\" (UniqueName: \"kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-token\") pod \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097440    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-socket\") pod \"bffcbfcb-f8c9-441f-a456-b55829cad058\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097499    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-envoy\") pod \"bffcbfcb-f8c9-441f-a456-b55829cad058\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097543    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-credential-socket\") pod \"453867f4-909e-4d42-8ac8-6ab6da4f0626\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097573    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-certs\") pod \"453867f4-909e-4d42-8ac8-6ab6da4f0626\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097608    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-podinfo\") pod \"bffcbfcb-f8c9-441f-a456-b55829cad058\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097699    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-kube-api-access-98fqc\" (UniqueName: \"kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-kube-api-access-98fqc\") pod \"bffcbfcb-f8c9-441f-a456-b55829cad058\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097730    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-envoy\") pod \"453867f4-909e-4d42-8ac8-6ab6da4f0626\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097761    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453867f4-909e-4d42-8ac8-6ab6da4f0626-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453867f4-909e-4d42-8ac8-6ab6da4f0626-istiod-ca-cert\") pod \"453867f4-909e-4d42-8ac8-6ab6da4f0626\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097790    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-certs\") pod \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097919    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-podinfo\") pod \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097966    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-token\" (UniqueName: \"kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-token\") pod \"453867f4-909e-4d42-8ac8-6ab6da4f0626\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.097995    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-credential-socket\") pod \"bffcbfcb-f8c9-441f-a456-b55829cad058\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.098117    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bffcbfcb-f8c9-441f-a456-b55829cad058-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffcbfcb-f8c9-441f-a456-b55829cad058-istiod-ca-cert\") pod \"bffcbfcb-f8c9-441f-a456-b55829cad058\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.098161    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-data\") pod \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.098204    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istiod-ca-cert\") pod \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.098241    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-certs\") pod \"bffcbfcb-f8c9-441f-a456-b55829cad058\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") "
Oct 29 23:00:13.099517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.098275    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-credential-socket\") pod \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") "
Oct 29 23:00:13.101150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.099966    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-credential-socket" pod "bffcbfcb-f8c9-441f-a456-b55829cad058" (UID: "bffcbfcb-f8c9-441f-a456-b55829cad058"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.101150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.100203    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-envoy\") pod \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") "
Oct 29 23:00:13.101150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.100283    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-socket\") pod \"453867f4-909e-4d42-8ac8-6ab6da4f0626\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") "
Oct 29 23:00:13.101150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.100345    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-data\") pod \"453867f4-909e-4d42-8ac8-6ab6da4f0626\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") "
Oct 29 23:00:13.101150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.100387    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-kube-api-access-gcr6s\" (UniqueName: \"kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-kube-api-access-gcr6s\") pod \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\" (UID: \"7c7b50c2-573d-41c0-b372-b51fd1e29ba6\") "
Oct 29 23:00:13.101150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.100419    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-kube-api-access-kwnp8\" (UniqueName: \"kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-kube-api-access-kwnp8\") pod \"453867f4-909e-4d42-8ac8-6ab6da4f0626\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") "
Oct 29 23:00:13.101150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.100437    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453867f4-909e-4d42-8ac8-6ab6da4f0626-istiod-ca-cert" pod "453867f4-909e-4d42-8ac8-6ab6da4f0626" (UID: "453867f4-909e-4d42-8ac8-6ab6da4f0626"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:13.101150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.100490    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-podinfo\") pod \"453867f4-909e-4d42-8ac8-6ab6da4f0626\" (UID: \"453867f4-909e-4d42-8ac8-6ab6da4f0626\") "
Oct 29 23:00:13.101150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.100518    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-data\") pod \"bffcbfcb-f8c9-441f-a456-b55829cad058\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") "
Oct 29 23:00:13.101150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.100596    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-token\" (UniqueName: \"kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-token\") pod \"bffcbfcb-f8c9-441f-a456-b55829cad058\" (UID: \"bffcbfcb-f8c9-441f-a456-b55829cad058\") "
Oct 29 23:00:13.101150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.100765    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-certs" pod "7c7b50c2-573d-41c0-b372-b51fd1e29ba6" (UID: "7c7b50c2-573d-41c0-b372-b51fd1e29ba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.101799 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.101549    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453867f4-909e-4d42-8ac8-6ab6da4f0626-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.101799 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.101572    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.101799 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.101584    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.121034 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.120978    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-token" pod "453867f4-909e-4d42-8ac8-6ab6da4f0626" (UID: "453867f4-909e-4d42-8ac8-6ab6da4f0626"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.121649 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.121198    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-socket" pod "7c7b50c2-573d-41c0-b372-b51fd1e29ba6" (UID: "7c7b50c2-573d-41c0-b372-b51fd1e29ba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.122159 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.122127    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-credential-socket" pod "453867f4-909e-4d42-8ac8-6ab6da4f0626" (UID: "453867f4-909e-4d42-8ac8-6ab6da4f0626"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.123103 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.123076    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bffcbfcb-f8c9-441f-a456-b55829cad058-istiod-ca-cert" pod "bffcbfcb-f8c9-441f-a456-b55829cad058" (UID: "bffcbfcb-f8c9-441f-a456-b55829cad058"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:13.123335 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.123301    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-data" pod "7c7b50c2-573d-41c0-b372-b51fd1e29ba6" (UID: "7c7b50c2-573d-41c0-b372-b51fd1e29ba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.123702 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.123680    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istiod-ca-cert" pod "7c7b50c2-573d-41c0-b372-b51fd1e29ba6" (UID: "7c7b50c2-573d-41c0-b372-b51fd1e29ba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:13.123978 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.123957    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-certs" pod "bffcbfcb-f8c9-441f-a456-b55829cad058" (UID: "bffcbfcb-f8c9-441f-a456-b55829cad058"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.124151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.124133    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-credential-socket" pod "7c7b50c2-573d-41c0-b372-b51fd1e29ba6" (UID: "7c7b50c2-573d-41c0-b372-b51fd1e29ba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.140348 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.127685    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-socket" pod "453867f4-909e-4d42-8ac8-6ab6da4f0626" (UID: "453867f4-909e-4d42-8ac8-6ab6da4f0626"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.140348 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.127876    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-data" pod "453867f4-909e-4d42-8ac8-6ab6da4f0626" (UID: "453867f4-909e-4d42-8ac8-6ab6da4f0626"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.140348 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.129550    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-certs" pod "453867f4-909e-4d42-8ac8-6ab6da4f0626" (UID: "453867f4-909e-4d42-8ac8-6ab6da4f0626"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.149668 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.141922    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-token" pod "7c7b50c2-573d-41c0-b372-b51fd1e29ba6" (UID: "7c7b50c2-573d-41c0-b372-b51fd1e29ba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.149668 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.142066    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-podinfo" pod "453867f4-909e-4d42-8ac8-6ab6da4f0626" (UID: "453867f4-909e-4d42-8ac8-6ab6da4f0626"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:13.149668 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.142176    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-socket" pod "bffcbfcb-f8c9-441f-a456-b55829cad058" (UID: "bffcbfcb-f8c9-441f-a456-b55829cad058"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.149668 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.142251    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-envoy" pod "7c7b50c2-573d-41c0-b372-b51fd1e29ba6" (UID: "7c7b50c2-573d-41c0-b372-b51fd1e29ba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.149668 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.142317    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-podinfo" pod "7c7b50c2-573d-41c0-b372-b51fd1e29ba6" (UID: "7c7b50c2-573d-41c0-b372-b51fd1e29ba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:13.149668 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.142606    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-data" pod "bffcbfcb-f8c9-441f-a456-b55829cad058" (UID: "bffcbfcb-f8c9-441f-a456-b55829cad058"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.149668 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.147811    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-kube-api-access-98fqc" pod "bffcbfcb-f8c9-441f-a456-b55829cad058" (UID: "bffcbfcb-f8c9-441f-a456-b55829cad058"). InnerVolumeSpecName "kube-api-access-98fqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.152922 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.152871    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:13.164663 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.159701    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-envoy" pod "bffcbfcb-f8c9-441f-a456-b55829cad058" (UID: "bffcbfcb-f8c9-441f-a456-b55829cad058"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.164663 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.164079    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-envoy" pod "453867f4-909e-4d42-8ac8-6ab6da4f0626" (UID: "453867f4-909e-4d42-8ac8-6ab6da4f0626"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:13.164663 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.164144    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-podinfo" pod "bffcbfcb-f8c9-441f-a456-b55829cad058" (UID: "bffcbfcb-f8c9-441f-a456-b55829cad058"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:13.164663 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.164147    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-kube-api-access-gcr6s" pod "7c7b50c2-573d-41c0-b372-b51fd1e29ba6" (UID: "7c7b50c2-573d-41c0-b372-b51fd1e29ba6"). InnerVolumeSpecName "kube-api-access-gcr6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.164663 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.164214    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-token" pod "bffcbfcb-f8c9-441f-a456-b55829cad058" (UID: "bffcbfcb-f8c9-441f-a456-b55829cad058"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.165310 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.165206    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-kube-api-access-kwnp8" pod "453867f4-909e-4d42-8ac8-6ab6da4f0626" (UID: "453867f4-909e-4d42-8ac8-6ab6da4f0626"). InnerVolumeSpecName "kube-api-access-kwnp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.203229 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203203    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203229 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203230    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203243    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203255    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203265    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203278    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203292    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203305    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203318    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98fqc\" (UniqueName: \"kubernetes.io/projected/bffcbfcb-f8c9-441f-a456-b55829cad058-kube-api-access-98fqc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203346    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203359    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203371    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203385    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffcbfcb-f8c9-441f-a456-b55829cad058-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203396    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203409    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203421    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203434    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203447    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203461    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203475    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.203482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203489    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcr6s\" (UniqueName: \"kubernetes.io/projected/7c7b50c2-573d-41c0-b372-b51fd1e29ba6-kube-api-access-gcr6s\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.204081 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203503    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwnp8\" (UniqueName: \"kubernetes.io/projected/453867f4-909e-4d42-8ac8-6ab6da4f0626-kube-api-access-kwnp8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.204081 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203516    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453867f4-909e-4d42-8ac8-6ab6da4f0626-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.204081 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.203530    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffcbfcb-f8c9-441f-a456-b55829cad058-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.490111 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.490081    4860 generic.go:356] "Generic (PLEG): container finished" podID="bffcbfcb-f8c9-441f-a456-b55829cad058" containerID="5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58" exitCode=0
Oct 29 23:00:13.490344 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.490130    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"
Oct 29 23:00:13.490344 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.490166    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv" event={"ID":"bffcbfcb-f8c9-441f-a456-b55829cad058","Type":"ContainerDied","Data":"5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58"}
Oct 29 23:00:13.490344 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.490225    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv" event={"ID":"bffcbfcb-f8c9-441f-a456-b55829cad058","Type":"ContainerDied","Data":"25069c08207452a83899f6b4ac7a9529058e96d6ec4dc374d186be9fb0bc29e5"}
Oct 29 23:00:13.490344 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.490255    4860 scope.go:122] "RemoveContainer" containerID="5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58"
Oct 29 23:00:13.491851 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.491832    4860 generic.go:356] "Generic (PLEG): container finished" podID="7c7b50c2-573d-41c0-b372-b51fd1e29ba6" containerID="20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878" exitCode=0
Oct 29 23:00:13.491970 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.491884    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"
Oct 29 23:00:13.491970 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.491923    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b" event={"ID":"7c7b50c2-573d-41c0-b372-b51fd1e29ba6","Type":"ContainerDied","Data":"20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878"}
Oct 29 23:00:13.492089 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.491976    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b" event={"ID":"7c7b50c2-573d-41c0-b372-b51fd1e29ba6","Type":"ContainerDied","Data":"c13c43be018afafae18507e4092cc6d1e8a815ce75f158e3e324be04b4d639a1"}
Oct 29 23:00:13.501228 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.501204    4860 generic.go:356] "Generic (PLEG): container finished" podID="453867f4-909e-4d42-8ac8-6ab6da4f0626" containerID="ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b" exitCode=0
Oct 29 23:00:13.501384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.501252    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"
Oct 29 23:00:13.501477 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.501456    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm" event={"ID":"453867f4-909e-4d42-8ac8-6ab6da4f0626","Type":"ContainerDied","Data":"ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b"}
Oct 29 23:00:13.501540 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.501502    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm" event={"ID":"453867f4-909e-4d42-8ac8-6ab6da4f0626","Type":"ContainerDied","Data":"23e2b0fbfcf7cfa517065bf86185c5eabfb48d94d5e557f2fe0e0330d27138e5"}
Oct 29 23:00:13.508760 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.508671    4860 scope.go:122] "RemoveContainer" containerID="5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58"
Oct 29 23:00:13.509191 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:13.509041    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58\": container with ID starting with 5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58 not found: ID does not exist" containerID="5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58"
Oct 29 23:00:13.509191 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.509075    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58"} err="failed to get container status \"5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58\": rpc error: code = NotFound desc = could not find container \"5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58\": container with ID starting with 5b70e6c5be94bc0efaba5bc5517d876f92f2f9fb7289819f5971b5fc8502ec58 not found: ID does not exist"
Oct 29 23:00:13.509191 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.509100    4860 scope.go:122] "RemoveContainer" containerID="20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878"
Oct 29 23:00:13.524285 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.524264    4860 scope.go:122] "RemoveContainer" containerID="20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878"
Oct 29 23:00:13.524583 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:13.524562    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878\": container with ID starting with 20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878 not found: ID does not exist" containerID="20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878"
Oct 29 23:00:13.524671 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.524593    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878"} err="failed to get container status \"20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878\": rpc error: code = NotFound desc = could not find container \"20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878\": container with ID starting with 20c9c3638801e55ee07dae38686f5d8a33ef07bbc594cb28f46fc8bae5e85878 not found: ID does not exist"
Oct 29 23:00:13.524671 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.524616    4860 scope.go:122] "RemoveContainer" containerID="ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b"
Oct 29 23:00:13.535929 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.535912    4860 scope.go:122] "RemoveContainer" containerID="ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b"
Oct 29 23:00:13.536199 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:13.536178    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b\": container with ID starting with ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b not found: ID does not exist" containerID="ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b"
Oct 29 23:00:13.536259 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.536209    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b"} err="failed to get container status \"ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b\": rpc error: code = NotFound desc = could not find container \"ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b\": container with ID starting with ba39d172d72198e7b0950edebe4e9c62255629aeb9204e2f336a0e71f773ca1b not found: ID does not exist"
Oct 29 23:00:13.552803 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.552781    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"]
Oct 29 23:00:13.566389 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.566361    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-ns4gv"]
Oct 29 23:00:13.617616 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.617593    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"]
Oct 29 23:00:13.622939 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.622914    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-vpr7b"]
Oct 29 23:00:13.668812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.668789    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"]
Oct 29 23:00:13.677269 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:13.677247    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f8ca96-openshift-default-7b49547d65-bn6sm"]
Oct 29 23:00:14.115578 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:14.115486    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453867f4-909e-4d42-8ac8-6ab6da4f0626" path="/var/lib/kubelet/pods/453867f4-909e-4d42-8ac8-6ab6da4f0626/volumes"
Oct 29 23:00:14.116084 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:14.116064    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7b50c2-573d-41c0-b372-b51fd1e29ba6" path="/var/lib/kubelet/pods/7c7b50c2-573d-41c0-b372-b51fd1e29ba6/volumes"
Oct 29 23:00:14.116567 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:14.116551    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bffcbfcb-f8c9-441f-a456-b55829cad058" path="/var/lib/kubelet/pods/bffcbfcb-f8c9-441f-a456-b55829cad058/volumes"
Oct 29 23:00:14.318281 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:14.318253    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"
Oct 29 23:00:14.628045 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:14.628012    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:14.977373 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:14.977309    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:15.356270 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:15.356199    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"
Oct 29 23:00:15.446645 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:15.446616    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:00:15.481468 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:15.481423    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:16.195132 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:16.195100    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:17.104350 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:17.104163    4860 scope.go:122] "RemoveContainer" containerID="44c90ab611df18c2b7a2787656aad9de8896a3f5ab3b8879f7c1d6667b03375c"
Oct 29 23:00:17.104806 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:17.104391    4860 scope.go:122] "RemoveContainer" containerID="3549fb7a48e4a5b4d0f3420e33cbb8958737b99252afb4629d5abe56895e875b"
Oct 29 23:00:17.105042 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:17.105020    4860 scope.go:122] "RemoveContainer" containerID="b01db0d15e20cfc43f0da7809a68d4b85b5cb6ab35a7c4e006ab8a8a997b0f0b"
Oct 29 23:00:17.105579 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:17.105564    4860 scope.go:122] "RemoveContainer" containerID="42a43916634bee886ba292257bb235b1f14b6908239a0adb808bd6928569dfe3"
Oct 29 23:00:17.106105 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:17.106085    4860 scope.go:122] "RemoveContainer" containerID="054f657249b963eb4a424f5fae35a2edbac59211917a06f725decef74ded6db3"
Oct 29 23:00:17.732193 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:17.732153    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:17.852612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:17.852581    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:17.987401 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:17.982722    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:18.162349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.157081    4860 scope.go:122] "RemoveContainer" containerID="97202d57693f929a0ab4035108776d2a9040d8d6036e9b1fb334ec2cc852e751"
Oct 29 23:00:18.162349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.159978    4860 scope.go:122] "RemoveContainer" containerID="6fab43b6b66ecaa3bedec595b768129a51ba60d67ff3cfbb412bda0cc37c692c"
Oct 29 23:00:18.162349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.160618    4860 scope.go:122] "RemoveContainer" containerID="a57ca4980909e1ae4971c6d5cf3ad2f188f39dbf535a2c52eadc1ad0340f06c1"
Oct 29 23:00:18.173666 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.165223    4860 scope.go:122] "RemoveContainer" containerID="3c4e44c7ebc9235e965ab382fb1b08f3d54fcb80b37f3d892feb2d642ff2d6d3"
Oct 29 23:00:18.211397 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:18.208947    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:00:18.443147 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.443109    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:18.584027 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.583985    4860 generic.go:356] "Generic (PLEG): container finished" podID="0b239203-2773-4ff8-bc28-604940a34500" containerID="a010ff289880274b9e3408ecac1cc2150ab45945335914aa0382303c374152ec" exitCode=60
Oct 29 23:00:18.584179 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.584061    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb" event={"ID":"0b239203-2773-4ff8-bc28-604940a34500","Type":"ContainerDied","Data":"a010ff289880274b9e3408ecac1cc2150ab45945335914aa0382303c374152ec"}
Oct 29 23:00:18.584236 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.584177    4860 scope.go:122] "RemoveContainer" containerID="42a43916634bee886ba292257bb235b1f14b6908239a0adb808bd6928569dfe3"
Oct 29 23:00:18.584950 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.584804    4860 scope.go:122] "RemoveContainer" containerID="a010ff289880274b9e3408ecac1cc2150ab45945335914aa0382303c374152ec"
Oct 29 23:00:18.585399 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:18.585317    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb_odh-ai-gateway-infra(0b239203-2773-4ff8-bc28-604940a34500)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb" podUID="0b239203-2773-4ff8-bc28-604940a34500"
Oct 29 23:00:18.588455 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.588429    4860 generic.go:356] "Generic (PLEG): container finished" podID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" containerID="2ab1d8f795a81ad6f5cf1e77eef48d73bd5e12d5933fd2cb0b2a6b37fbd2cf95" exitCode=60
Oct 29 23:00:18.588557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.588473    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86" event={"ID":"2f461145-f50b-4ebd-b9fa-25edb7d2af1a","Type":"ContainerDied","Data":"2ab1d8f795a81ad6f5cf1e77eef48d73bd5e12d5933fd2cb0b2a6b37fbd2cf95"}
Oct 29 23:00:18.588875 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.588859    4860 scope.go:122] "RemoveContainer" containerID="2ab1d8f795a81ad6f5cf1e77eef48d73bd5e12d5933fd2cb0b2a6b37fbd2cf95"
Oct 29 23:00:18.589261 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:18.589239    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86_odh-ai-gateway-infra(2f461145-f50b-4ebd-b9fa-25edb7d2af1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a"
Oct 29 23:00:18.590987 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.590968    4860 generic.go:356] "Generic (PLEG): container finished" podID="32dbf81d-0294-4922-b785-1a0b3950017a" containerID="9abde9cdfcfeab985b20970c3ec92f286728979dbba3c7bd0f55349487fbd8e0" exitCode=60
Oct 29 23:00:18.591078 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.591047    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd" event={"ID":"32dbf81d-0294-4922-b785-1a0b3950017a","Type":"ContainerDied","Data":"9abde9cdfcfeab985b20970c3ec92f286728979dbba3c7bd0f55349487fbd8e0"}
Oct 29 23:00:18.591518 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.591498    4860 scope.go:122] "RemoveContainer" containerID="9abde9cdfcfeab985b20970c3ec92f286728979dbba3c7bd0f55349487fbd8e0"
Oct 29 23:00:18.591921 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:18.591896    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd_odh-ai-gateway-infra(32dbf81d-0294-4922-b785-1a0b3950017a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd" podUID="32dbf81d-0294-4922-b785-1a0b3950017a"
Oct 29 23:00:18.593186 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.593167    4860 generic.go:356] "Generic (PLEG): container finished" podID="2247bdec-f506-4144-a05a-884528c5c132" containerID="f4f56280f102afb0c7a2fd20212e680fd066d87cf6884ca90db7fedad09ec9ba" exitCode=60
Oct 29 23:00:18.593282 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.593233    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk" event={"ID":"2247bdec-f506-4144-a05a-884528c5c132","Type":"ContainerDied","Data":"f4f56280f102afb0c7a2fd20212e680fd066d87cf6884ca90db7fedad09ec9ba"}
Oct 29 23:00:18.593596 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.593580    4860 scope.go:122] "RemoveContainer" containerID="f4f56280f102afb0c7a2fd20212e680fd066d87cf6884ca90db7fedad09ec9ba"
Oct 29 23:00:18.593882 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:18.593863    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk_odh-ai-gateway-infra(2247bdec-f506-4144-a05a-884528c5c132)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk" podUID="2247bdec-f506-4144-a05a-884528c5c132"
Oct 29 23:00:18.595271 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.595248    4860 generic.go:356] "Generic (PLEG): container finished" podID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" containerID="3fd8c040757c928fc873c9bb77dbefc48b40932d6acf9049c923425304c3ab98" exitCode=60
Oct 29 23:00:18.595414 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.595318    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6" event={"ID":"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76","Type":"ContainerDied","Data":"3fd8c040757c928fc873c9bb77dbefc48b40932d6acf9049c923425304c3ab98"}
Oct 29 23:00:18.595646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.595630    4860 scope.go:122] "RemoveContainer" containerID="3fd8c040757c928fc873c9bb77dbefc48b40932d6acf9049c923425304c3ab98"
Oct 29 23:00:18.595890 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:18.595873    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6_odh-ai-gateway-infra(e27bd901-d6a1-48c3-91fe-b9c12b3d7b76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76"
Oct 29 23:00:18.631857 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.631836    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:18.657747 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.657727    4860 scope.go:122] "RemoveContainer" containerID="054f657249b963eb4a424f5fae35a2edbac59211917a06f725decef74ded6db3"
Oct 29 23:00:18.683957 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.683930    4860 scope.go:122] "RemoveContainer" containerID="44c90ab611df18c2b7a2787656aad9de8896a3f5ab3b8879f7c1d6667b03375c"
Oct 29 23:00:18.699589 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.699564    4860 scope.go:122] "RemoveContainer" containerID="b01db0d15e20cfc43f0da7809a68d4b85b5cb6ab35a7c4e006ab8a8a997b0f0b"
Oct 29 23:00:18.718826 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:18.718805    4860 scope.go:122] "RemoveContainer" containerID="3549fb7a48e4a5b4d0f3420e33cbb8958737b99252afb4629d5abe56895e875b"
Oct 29 23:00:19.112348 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.107968    4860 scope.go:122] "RemoveContainer" containerID="264fc49da7b2a82795f866e48cd9a0c91f1584c85c229db749acef5fb4538992"
Oct 29 23:00:19.112348 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.108723    4860 scope.go:122] "RemoveContainer" containerID="152fe06aa13b47948b2c08000df8be4cf933e068a37f8201accfd095d65f7a38"
Oct 29 23:00:19.536142 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.536106    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:19.635571 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.635541    4860 generic.go:356] "Generic (PLEG): container finished" podID="4e276a4c-e944-495d-8d4e-f80ebd436f05" containerID="1e58aac2ef4295e18fd6848d33d623fea725339ee4a1ecdda5dea57150df00f8" exitCode=60
Oct 29 23:00:19.635814 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.635605    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds" event={"ID":"4e276a4c-e944-495d-8d4e-f80ebd436f05","Type":"ContainerDied","Data":"1e58aac2ef4295e18fd6848d33d623fea725339ee4a1ecdda5dea57150df00f8"}
Oct 29 23:00:19.635911 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.635852    4860 scope.go:122] "RemoveContainer" containerID="a57ca4980909e1ae4971c6d5cf3ad2f188f39dbf535a2c52eadc1ad0340f06c1"
Oct 29 23:00:19.636415 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.636390    4860 scope.go:122] "RemoveContainer" containerID="1e58aac2ef4295e18fd6848d33d623fea725339ee4a1ecdda5dea57150df00f8"
Oct 29 23:00:19.636809 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:19.636780    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds_odh-ai-gateway-infra(4e276a4c-e944-495d-8d4e-f80ebd436f05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05"
Oct 29 23:00:19.641451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.640557    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks" event={"ID":"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c","Type":"ContainerStarted","Data":"a07dafaa18b9a8b73e43c53f8b02fab67bb59016338a76f3cfc7c6c7161353bb"}
Oct 29 23:00:19.645686 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.645653    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl" event={"ID":"521dba0b-944b-4ee0-8e00-60fa687c94bd","Type":"ContainerStarted","Data":"305277bf5f0c396ebe6a9c5a0f0d82143e20c0aaefeb8968df7ea7973b0bc809"}
Oct 29 23:00:19.647783 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.647760    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k" event={"ID":"b151b922-7435-4d4e-a403-3c911de5d8f6","Type":"ContainerStarted","Data":"d3f808c60e611f12403a7acc16e9e0db409e9ee954a10b680ec0a4fa9767b781"}
Oct 29 23:00:19.665980 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.665906    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:19.671935 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.671105    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"]
Oct 29 23:00:19.721476 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.721441    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"]
Oct 29 23:00:19.773643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.771539    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"]
Oct 29 23:00:19.813267 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.812722    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:19.831890 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.831858    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"]
Oct 29 23:00:19.857387 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.857096    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"]
Oct 29 23:00:19.886999 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.886969    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"]
Oct 29 23:00:19.887276 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.887202    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6" podUID="55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" containerName="istio-proxy" containerID="cri-o://ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02" gracePeriod=30
Oct 29 23:00:19.958812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:19.958775    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"
Oct 29 23:00:20.132463 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.115061    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:20.132463 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.131852    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b239203-2773-4ff8-bc28-604940a34500-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b239203-2773-4ff8-bc28-604940a34500-maas-api-service-ca\") pod \"0b239203-2773-4ff8-bc28-604940a34500\" (UID: \"0b239203-2773-4ff8-bc28-604940a34500\") "
Oct 29 23:00:20.132463 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.132216    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b239203-2773-4ff8-bc28-604940a34500-kube-api-access-r8lnj\" (UniqueName: \"kubernetes.io/projected/0b239203-2773-4ff8-bc28-604940a34500-kube-api-access-r8lnj\") pod \"0b239203-2773-4ff8-bc28-604940a34500\" (UID: \"0b239203-2773-4ff8-bc28-604940a34500\") "
Oct 29 23:00:20.143531 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.142049    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b239203-2773-4ff8-bc28-604940a34500-maas-api-service-ca" pod "0b239203-2773-4ff8-bc28-604940a34500" (UID: "0b239203-2773-4ff8-bc28-604940a34500"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:20.237407 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.237371    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b239203-2773-4ff8-bc28-604940a34500-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.255670 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.255636    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b239203-2773-4ff8-bc28-604940a34500-kube-api-access-r8lnj" pod "0b239203-2773-4ff8-bc28-604940a34500" (UID: "0b239203-2773-4ff8-bc28-604940a34500"). InnerVolumeSpecName "kube-api-access-r8lnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.330215 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.330180    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"]
Oct 29 23:00:20.331184 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331152    4860 cpu_manager.go:414] "containerMap: removing container" podUID="bffcbfcb-f8c9-441f-a456-b55829cad058" containerName="istio-proxy"
Oct 29 23:00:20.331184 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331176    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffcbfcb-f8c9-441f-a456-b55829cad058" containerName="istio-proxy"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331224    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4aaed535-f4bd-430d-8528-af2feb5a8ca4" containerName="payload-pre-processing"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331234    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aaed535-f4bd-430d-8528-af2feb5a8ca4" containerName="payload-pre-processing"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331263    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f5e951ea-8eb5-4a3a-aabf-dde45406c921" containerName="payload-processing"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331270    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e951ea-8eb5-4a3a-aabf-dde45406c921" containerName="payload-processing"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331281    4860 cpu_manager.go:414] "containerMap: removing container" podUID="7c7b50c2-573d-41c0-b372-b51fd1e29ba6" containerName="istio-proxy"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331289    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7b50c2-573d-41c0-b372-b51fd1e29ba6" containerName="istio-proxy"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331307    4860 cpu_manager.go:414] "containerMap: removing container" podUID="cb80ca82-acc6-4bdf-b76a-f39eb983f49a" containerName="cleanup"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331315    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb80ca82-acc6-4bdf-b76a-f39eb983f49a" containerName="cleanup"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331347    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0b239203-2773-4ff8-bc28-604940a34500" containerName="cleanup"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331354    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b239203-2773-4ff8-bc28-604940a34500" containerName="cleanup"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331364    4860 cpu_manager.go:414] "containerMap: removing container" podUID="453867f4-909e-4d42-8ac8-6ab6da4f0626" containerName="istio-proxy"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331372    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453867f4-909e-4d42-8ac8-6ab6da4f0626" containerName="istio-proxy"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331388    4860 cpu_manager.go:414] "containerMap: removing container" podUID="cb80ca82-acc6-4bdf-b76a-f39eb983f49a" containerName="cleanup"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331395    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb80ca82-acc6-4bdf-b76a-f39eb983f49a" containerName="cleanup"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331444    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0b239203-2773-4ff8-bc28-604940a34500" containerName="cleanup"
Oct 29 23:00:20.331448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331451    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b239203-2773-4ff8-bc28-604940a34500" containerName="cleanup"
Oct 29 23:00:20.332300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331728    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb80ca82-acc6-4bdf-b76a-f39eb983f49a" containerName="cleanup"
Oct 29 23:00:20.332300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331748    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b239203-2773-4ff8-bc28-604940a34500" containerName="cleanup"
Oct 29 23:00:20.332300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331762    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b239203-2773-4ff8-bc28-604940a34500" containerName="cleanup"
Oct 29 23:00:20.332300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331774    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffcbfcb-f8c9-441f-a456-b55829cad058" containerName="istio-proxy"
Oct 29 23:00:20.332300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331794    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aaed535-f4bd-430d-8528-af2feb5a8ca4" containerName="payload-pre-processing"
Oct 29 23:00:20.332300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331810    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e951ea-8eb5-4a3a-aabf-dde45406c921" containerName="payload-processing"
Oct 29 23:00:20.332300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331820    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7b50c2-573d-41c0-b372-b51fd1e29ba6" containerName="istio-proxy"
Oct 29 23:00:20.332300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331831    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b239203-2773-4ff8-bc28-604940a34500" containerName="cleanup"
Oct 29 23:00:20.332300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.331842    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="453867f4-909e-4d42-8ac8-6ab6da4f0626" containerName="istio-proxy"
Oct 29 23:00:20.335913 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.335894    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:20.342705 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.342674    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8lnj\" (UniqueName: \"kubernetes.io/projected/0b239203-2773-4ff8-bc28-604940a34500-kube-api-access-r8lnj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.354141 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.354101    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:20.358978 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.358954    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"]
Oct 29 23:00:20.386043 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.386019    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"
Oct 29 23:00:20.395500 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.395474    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"
Oct 29 23:00:20.444435 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.444405    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-maas-api-service-ca\") pod \"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76\" (UID: \"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76\") "
Oct 29 23:00:20.444652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.444631    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-maas-api-service-ca\") pod \"2f461145-f50b-4ebd-b9fa-25edb7d2af1a\" (UID: \"2f461145-f50b-4ebd-b9fa-25edb7d2af1a\") "
Oct 29 23:00:20.445147 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.445124    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-kube-api-access-g7k7m\" (UniqueName: \"kubernetes.io/projected/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-kube-api-access-g7k7m\") pod \"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76\" (UID: \"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76\") "
Oct 29 23:00:20.445714 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.445695    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-kube-api-access-76pns\" (UniqueName: \"kubernetes.io/projected/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-kube-api-access-76pns\") pod \"2f461145-f50b-4ebd-b9fa-25edb7d2af1a\" (UID: \"2f461145-f50b-4ebd-b9fa-25edb7d2af1a\") "
Oct 29 23:00:20.446535 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.446513    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc\" (UID: \"ec6cc782-8039-4b69-af59-053bc06853ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:20.447254 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.446714    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj8jj\" (UniqueName: \"kubernetes.io/projected/ec6cc782-8039-4b69-af59-053bc06853ce-kube-api-access-zj8jj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc\" (UID: \"ec6cc782-8039-4b69-af59-053bc06853ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:20.447254 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.446795    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc\" (UID: \"ec6cc782-8039-4b69-af59-053bc06853ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:20.448657 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.448620    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-maas-api-service-ca" pod "e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" (UID: "e27bd901-d6a1-48c3-91fe-b9c12b3d7b76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:20.450771 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.450741    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-maas-api-service-ca" pod "2f461145-f50b-4ebd-b9fa-25edb7d2af1a" (UID: "2f461145-f50b-4ebd-b9fa-25edb7d2af1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:20.454199 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.454164    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-kube-api-access-g7k7m" pod "e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" (UID: "e27bd901-d6a1-48c3-91fe-b9c12b3d7b76"). InnerVolumeSpecName "kube-api-access-g7k7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.473270 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.473108    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-kube-api-access-76pns" pod "2f461145-f50b-4ebd-b9fa-25edb7d2af1a" (UID: "2f461145-f50b-4ebd-b9fa-25edb7d2af1a"). InnerVolumeSpecName "kube-api-access-76pns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.547981 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.547952    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc\" (UID: \"ec6cc782-8039-4b69-af59-053bc06853ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:20.548431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.547991    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj8jj\" (UniqueName: \"kubernetes.io/projected/ec6cc782-8039-4b69-af59-053bc06853ce-kube-api-access-zj8jj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc\" (UID: \"ec6cc782-8039-4b69-af59-053bc06853ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:20.548431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.548032    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc\" (UID: \"ec6cc782-8039-4b69-af59-053bc06853ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:20.548431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.548274    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7k7m\" (UniqueName: \"kubernetes.io/projected/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-kube-api-access-g7k7m\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.548431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.548287    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76pns\" (UniqueName: \"kubernetes.io/projected/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-kube-api-access-76pns\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.548431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.548297    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.548431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.548305    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f461145-f50b-4ebd-b9fa-25edb7d2af1a-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.548870 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.548845    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc\" (UID: \"ec6cc782-8039-4b69-af59-053bc06853ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:20.550734 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.550710    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc\" (UID: \"ec6cc782-8039-4b69-af59-053bc06853ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:20.569692 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.569667    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj8jj\" (UniqueName: \"kubernetes.io/projected/ec6cc782-8039-4b69-af59-053bc06853ce-kube-api-access-zj8jj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc\" (UID: \"ec6cc782-8039-4b69-af59-053bc06853ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:20.599969 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.599950    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"
Oct 29 23:00:20.603473 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.603452    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"
Oct 29 23:00:20.649411 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.649381    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2247bdec-f506-4144-a05a-884528c5c132-kube-api-access-hk56g\" (UniqueName: \"kubernetes.io/projected/2247bdec-f506-4144-a05a-884528c5c132-kube-api-access-hk56g\") pod \"2247bdec-f506-4144-a05a-884528c5c132\" (UID: \"2247bdec-f506-4144-a05a-884528c5c132\") "
Oct 29 23:00:20.649543 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.649477    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32dbf81d-0294-4922-b785-1a0b3950017a-kube-api-access-vmnbf\" (UniqueName: \"kubernetes.io/projected/32dbf81d-0294-4922-b785-1a0b3950017a-kube-api-access-vmnbf\") pod \"32dbf81d-0294-4922-b785-1a0b3950017a\" (UID: \"32dbf81d-0294-4922-b785-1a0b3950017a\") "
Oct 29 23:00:20.649879 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.649857    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32dbf81d-0294-4922-b785-1a0b3950017a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32dbf81d-0294-4922-b785-1a0b3950017a-maas-api-service-ca\") pod \"32dbf81d-0294-4922-b785-1a0b3950017a\" (UID: \"32dbf81d-0294-4922-b785-1a0b3950017a\") "
Oct 29 23:00:20.649966 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.649928    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2247bdec-f506-4144-a05a-884528c5c132-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2247bdec-f506-4144-a05a-884528c5c132-maas-api-service-ca\") pod \"2247bdec-f506-4144-a05a-884528c5c132\" (UID: \"2247bdec-f506-4144-a05a-884528c5c132\") "
Oct 29 23:00:20.651209 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.651185    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2247bdec-f506-4144-a05a-884528c5c132-maas-api-service-ca" pod "2247bdec-f506-4144-a05a-884528c5c132" (UID: "2247bdec-f506-4144-a05a-884528c5c132"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:20.651389 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.651273    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32dbf81d-0294-4922-b785-1a0b3950017a-maas-api-service-ca" pod "32dbf81d-0294-4922-b785-1a0b3950017a" (UID: "32dbf81d-0294-4922-b785-1a0b3950017a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:20.658305 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.658280    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:20.667043 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.667015    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk" event={"ID":"2247bdec-f506-4144-a05a-884528c5c132","Type":"ContainerDied","Data":"57fad2b7cda15b287ab5191393fd3fd10e6f09dd45bd1f483fe46116ed1d40c4"}
Oct 29 23:00:20.667148 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.667080    4860 scope.go:122] "RemoveContainer" containerID="f4f56280f102afb0c7a2fd20212e680fd066d87cf6884ca90db7fedad09ec9ba"
Oct 29 23:00:20.667224 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.667197    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"
Oct 29 23:00:20.671587 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.670910    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4" event={"ID":"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d","Type":"ContainerStarted","Data":"88626d8725a43348b53069880d111b92d1ef2fe23f11c91cd711bc6c19b448b9"}
Oct 29 23:00:20.671587 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.670913    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2247bdec-f506-4144-a05a-884528c5c132-kube-api-access-hk56g" pod "2247bdec-f506-4144-a05a-884528c5c132" (UID: "2247bdec-f506-4144-a05a-884528c5c132"). InnerVolumeSpecName "kube-api-access-hk56g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.676172 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.676145    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6" event={"ID":"e27bd901-d6a1-48c3-91fe-b9c12b3d7b76","Type":"ContainerDied","Data":"b242a13d025b97920cc88199beef746cd16ad9f4f8c0b4df2317891e7789bbc2"}
Oct 29 23:00:20.676301 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.676184    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"
Oct 29 23:00:20.679112 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.679072    4860 generic.go:356] "Generic (PLEG): container finished" podID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" containerID="a07dafaa18b9a8b73e43c53f8b02fab67bb59016338a76f3cfc7c6c7161353bb" exitCode=60
Oct 29 23:00:20.679227 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.679209    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks" event={"ID":"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c","Type":"ContainerDied","Data":"a07dafaa18b9a8b73e43c53f8b02fab67bb59016338a76f3cfc7c6c7161353bb"}
Oct 29 23:00:20.679699 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.679677    4860 scope.go:122] "RemoveContainer" containerID="a07dafaa18b9a8b73e43c53f8b02fab67bb59016338a76f3cfc7c6c7161353bb"
Oct 29 23:00:20.680222 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:20.680089    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks_odh-ai-gateway-infra(4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c"
Oct 29 23:00:20.681959 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.681933    4860 generic.go:356] "Generic (PLEG): container finished" podID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" containerID="b9952b3c710081a74381d8b6622aece60726a752d31fa5b8751715540dc5a269" exitCode=60
Oct 29 23:00:20.682050 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.682022    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n" event={"ID":"7c7a942f-a12b-42b1-8b11-b6e2c5103798","Type":"ContainerDied","Data":"b9952b3c710081a74381d8b6622aece60726a752d31fa5b8751715540dc5a269"}
Oct 29 23:00:20.682443 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.682418    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32dbf81d-0294-4922-b785-1a0b3950017a-kube-api-access-vmnbf" pod "32dbf81d-0294-4922-b785-1a0b3950017a" (UID: "32dbf81d-0294-4922-b785-1a0b3950017a"). InnerVolumeSpecName "kube-api-access-vmnbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.682546 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.682528    4860 scope.go:122] "RemoveContainer" containerID="b9952b3c710081a74381d8b6622aece60726a752d31fa5b8751715540dc5a269"
Oct 29 23:00:20.683750 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:20.683686    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n_odh-ai-gateway-infra(7c7a942f-a12b-42b1-8b11-b6e2c5103798)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798"
Oct 29 23:00:20.691369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.691301    4860 generic.go:356] "Generic (PLEG): container finished" podID="521dba0b-944b-4ee0-8e00-60fa687c94bd" containerID="305277bf5f0c396ebe6a9c5a0f0d82143e20c0aaefeb8968df7ea7973b0bc809" exitCode=60
Oct 29 23:00:20.691524 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.691503    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl" event={"ID":"521dba0b-944b-4ee0-8e00-60fa687c94bd","Type":"ContainerDied","Data":"305277bf5f0c396ebe6a9c5a0f0d82143e20c0aaefeb8968df7ea7973b0bc809"}
Oct 29 23:00:20.692338 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.691928    4860 scope.go:122] "RemoveContainer" containerID="305277bf5f0c396ebe6a9c5a0f0d82143e20c0aaefeb8968df7ea7973b0bc809"
Oct 29 23:00:20.692338 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:20.692269    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl_odh-ai-gateway-infra(521dba0b-944b-4ee0-8e00-60fa687c94bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd"
Oct 29 23:00:20.695311 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.695149    4860 generic.go:356] "Generic (PLEG): container finished" podID="b151b922-7435-4d4e-a403-3c911de5d8f6" containerID="d3f808c60e611f12403a7acc16e9e0db409e9ee954a10b680ec0a4fa9767b781" exitCode=60
Oct 29 23:00:20.695447 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.695259    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k" event={"ID":"b151b922-7435-4d4e-a403-3c911de5d8f6","Type":"ContainerDied","Data":"d3f808c60e611f12403a7acc16e9e0db409e9ee954a10b680ec0a4fa9767b781"}
Oct 29 23:00:20.696381 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.695771    4860 scope.go:122] "RemoveContainer" containerID="d3f808c60e611f12403a7acc16e9e0db409e9ee954a10b680ec0a4fa9767b781"
Oct 29 23:00:20.696381 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:20.696025    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k_odh-ai-gateway-infra(b151b922-7435-4d4e-a403-3c911de5d8f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6"
Oct 29 23:00:20.698954 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.698928    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb" event={"ID":"0b239203-2773-4ff8-bc28-604940a34500","Type":"ContainerDied","Data":"30238ba32beea8b9a2a46ba022e604cd175571ad56c6e1b065fcaf384368bf64"}
Oct 29 23:00:20.699065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.699025    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"
Oct 29 23:00:20.700384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.700362    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"]
Oct 29 23:00:20.707420 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.704974    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86" event={"ID":"2f461145-f50b-4ebd-b9fa-25edb7d2af1a","Type":"ContainerDied","Data":"fc66ad5a21e31a4618d8c52ead6ce167d176fe7960c39cbe4931a900e190ebd7"}
Oct 29 23:00:20.707420 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.705068    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"
Oct 29 23:00:20.719338 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.719295    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd" event={"ID":"32dbf81d-0294-4922-b785-1a0b3950017a","Type":"ContainerDied","Data":"a0f0c5112311ca366c9e64e180a0591afd2dd0edca970412cd97bda2801159ad"}
Oct 29 23:00:20.720088 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.720066    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"
Oct 29 23:00:20.736371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.736042    4860 scope.go:122] "RemoveContainer" containerID="3fd8c040757c928fc873c9bb77dbefc48b40932d6acf9049c923425304c3ab98"
Oct 29 23:00:20.754282 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.754160    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32dbf81d-0294-4922-b785-1a0b3950017a-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.754282 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.754189    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2247bdec-f506-4144-a05a-884528c5c132-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.754282 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.754203    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk56g\" (UniqueName: \"kubernetes.io/projected/2247bdec-f506-4144-a05a-884528c5c132-kube-api-access-hk56g\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.754282 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.754218    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmnbf\" (UniqueName: \"kubernetes.io/projected/32dbf81d-0294-4922-b785-1a0b3950017a-kube-api-access-vmnbf\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.755157 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.755137    4860 scope.go:122] "RemoveContainer" containerID="3c4e44c7ebc9235e965ab382fb1b08f3d54fcb80b37f3d892feb2d642ff2d6d3"
Oct 29 23:00:20.757774 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.757598    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"]
Oct 29 23:00:20.788987 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.788963    4860 scope.go:122] "RemoveContainer" containerID="264fc49da7b2a82795f866e48cd9a0c91f1584c85c229db749acef5fb4538992"
Oct 29 23:00:20.808095 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.807988    4860 scope.go:122] "RemoveContainer" containerID="6fab43b6b66ecaa3bedec595b768129a51ba60d67ff3cfbb412bda0cc37c692c"
Oct 29 23:00:20.825987 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.825964    4860 scope.go:122] "RemoveContainer" containerID="97202d57693f929a0ab4035108776d2a9040d8d6036e9b1fb334ec2cc852e751"
Oct 29 23:00:20.852548 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.852449    4860 scope.go:122] "RemoveContainer" containerID="a010ff289880274b9e3408ecac1cc2150ab45945335914aa0382303c374152ec"
Oct 29 23:00:20.859795 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.859764    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"]
Oct 29 23:00:20.871815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.871788    4860 scope.go:122] "RemoveContainer" containerID="2ab1d8f795a81ad6f5cf1e77eef48d73bd5e12d5933fd2cb0b2a6b37fbd2cf95"
Oct 29 23:00:20.887227 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.887199    4860 scope.go:122] "RemoveContainer" containerID="9abde9cdfcfeab985b20970c3ec92f286728979dbba3c7bd0f55349487fbd8e0"
Oct 29 23:00:20.907411 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.907384    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"]
Oct 29 23:00:20.979783 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.979754    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"]
Oct 29 23:00:20.985606 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.985579    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c933cb-29888580-k8dh6"]
Oct 29 23:00:20.991247 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:20.991226    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"
Oct 29 23:00:20.999766 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:00:20.999740    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6cc782_8039_4b69_af59_053bc06853ce.slice/crio-bbb929646f60be6927b2db3bc4b77f8bc0333a39bc95c7381e7170a53e41269f WatchSource:0}: Error finding container bbb929646f60be6927b2db3bc4b77f8bc0333a39bc95c7381e7170a53e41269f: Status 404 returned error can't find the container with id bbb929646f60be6927b2db3bc4b77f8bc0333a39bc95c7381e7170a53e41269f
Oct 29 23:00:21.003682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.003657    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"]
Oct 29 23:00:21.059641 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.059613    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e276a4c-e944-495d-8d4e-f80ebd436f05-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e276a4c-e944-495d-8d4e-f80ebd436f05-maas-api-service-ca\") pod \"4e276a4c-e944-495d-8d4e-f80ebd436f05\" (UID: \"4e276a4c-e944-495d-8d4e-f80ebd436f05\") "
Oct 29 23:00:21.059797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.059786    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e276a4c-e944-495d-8d4e-f80ebd436f05-kube-api-access-nvmtz\" (UniqueName: \"kubernetes.io/projected/4e276a4c-e944-495d-8d4e-f80ebd436f05-kube-api-access-nvmtz\") pod \"4e276a4c-e944-495d-8d4e-f80ebd436f05\" (UID: \"4e276a4c-e944-495d-8d4e-f80ebd436f05\") "
Oct 29 23:00:21.059999 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.059973    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e276a4c-e944-495d-8d4e-f80ebd436f05-maas-api-service-ca" pod "4e276a4c-e944-495d-8d4e-f80ebd436f05" (UID: "4e276a4c-e944-495d-8d4e-f80ebd436f05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:21.060360 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.060305    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e276a4c-e944-495d-8d4e-f80ebd436f05-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:21.061769 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.061746    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e276a4c-e944-495d-8d4e-f80ebd436f05-kube-api-access-nvmtz" pod "4e276a4c-e944-495d-8d4e-f80ebd436f05" (UID: "4e276a4c-e944-495d-8d4e-f80ebd436f05"). InnerVolumeSpecName "kube-api-access-nvmtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:21.069956 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.069932    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"]
Oct 29 23:00:21.075297 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.075273    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b8f54c-29888580-z4qpb"]
Oct 29 23:00:21.113104 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.113078    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"]
Oct 29 23:00:21.118520 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.118497    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-920032-29888580-84j86"]
Oct 29 23:00:21.161462 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.161438    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvmtz\" (UniqueName: \"kubernetes.io/projected/4e276a4c-e944-495d-8d4e-f80ebd436f05-kube-api-access-nvmtz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:21.201160 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.201136    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"]
Oct 29 23:00:21.213497 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.213466    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-3bd0f6-29888580-b8hmd"]
Oct 29 23:00:21.318900 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.318779    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"]
Oct 29 23:00:21.323589 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.323563    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3ee017-29888580-llxxk"]
Oct 29 23:00:21.721168 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.721132    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"]
Oct 29 23:00:21.721976 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.721959    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" containerName="cleanup"
Oct 29 23:00:21.722018 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.721977    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" containerName="cleanup"
Oct 29 23:00:21.722018 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.721989    4860 cpu_manager.go:414] "containerMap: removing container" podUID="32dbf81d-0294-4922-b785-1a0b3950017a" containerName="cleanup"
Oct 29 23:00:21.722018 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.721994    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32dbf81d-0294-4922-b785-1a0b3950017a" containerName="cleanup"
Oct 29 23:00:21.722018 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722000    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" containerName="cleanup"
Oct 29 23:00:21.722018 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722006    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722029    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722034    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722041    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2247bdec-f506-4144-a05a-884528c5c132" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722046    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2247bdec-f506-4144-a05a-884528c5c132" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722057    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722061    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722074    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2247bdec-f506-4144-a05a-884528c5c132" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722078    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2247bdec-f506-4144-a05a-884528c5c132" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722088    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722092    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722102    4860 cpu_manager.go:414] "containerMap: removing container" podUID="32dbf81d-0294-4922-b785-1a0b3950017a" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722107    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32dbf81d-0294-4922-b785-1a0b3950017a" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722127    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0b239203-2773-4ff8-bc28-604940a34500" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722131    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b239203-2773-4ff8-bc28-604940a34500" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722140    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" containerName="cleanup"
Oct 29 23:00:21.722216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722147    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722280    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722296    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="32dbf81d-0294-4922-b785-1a0b3950017a" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722304    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2247bdec-f506-4144-a05a-884528c5c132" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722314    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722337    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722346    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722358    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="32dbf81d-0294-4922-b785-1a0b3950017a" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722364    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722375    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722380    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722386    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2247bdec-f506-4144-a05a-884528c5c132" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722397    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb80ca82-acc6-4bdf-b76a-f39eb983f49a" containerName="cleanup"
Oct 29 23:00:21.722805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.722409    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="32dbf81d-0294-4922-b785-1a0b3950017a" containerName="cleanup"
Oct 29 23:00:21.725669 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.725650    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:21.733452 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.733431    4860 generic.go:356] "Generic (PLEG): container finished" podID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" containerID="88626d8725a43348b53069880d111b92d1ef2fe23f11c91cd711bc6c19b448b9" exitCode=60
Oct 29 23:00:21.733563 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.733493    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4" event={"ID":"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d","Type":"ContainerDied","Data":"88626d8725a43348b53069880d111b92d1ef2fe23f11c91cd711bc6c19b448b9"}
Oct 29 23:00:21.733563 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.733537    4860 scope.go:122] "RemoveContainer" containerID="152fe06aa13b47948b2c08000df8be4cf933e068a37f8201accfd095d65f7a38"
Oct 29 23:00:21.735835 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.734569    4860 scope.go:122] "RemoveContainer" containerID="88626d8725a43348b53069880d111b92d1ef2fe23f11c91cd711bc6c19b448b9"
Oct 29 23:00:21.735835 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:21.734984    4860 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4_odh-ai-gateway-infra(bf8314b5-c567-4ebd-8a0b-d55cf6b3047d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d"
Oct 29 23:00:21.737604 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.736608    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds" event={"ID":"4e276a4c-e944-495d-8d4e-f80ebd436f05","Type":"ContainerDied","Data":"521e4df873de0192032eda112d1b235c2c6f6c035df1b81eb72aec7a5ed8e989"}
Oct 29 23:00:21.737604 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.736727    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"
Oct 29 23:00:21.747363 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.746638    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-63e89c-serving-cert\""
Oct 29 23:00:21.747882 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.747860    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"]
Oct 29 23:00:21.752544 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.752518    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc" event={"ID":"ec6cc782-8039-4b69-af59-053bc06853ce","Type":"ContainerStarted","Data":"55076b5bdb1943b44c97f28727e06b247d4b05d896957c94d637493c36f0b69f"}
Oct 29 23:00:21.752749 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.752721    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc" event={"ID":"ec6cc782-8039-4b69-af59-053bc06853ce","Type":"ContainerStarted","Data":"bbb929646f60be6927b2db3bc4b77f8bc0333a39bc95c7381e7170a53e41269f"}
Oct 29 23:00:21.759838 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.759817    4860 scope.go:122] "RemoveContainer" containerID="1e58aac2ef4295e18fd6848d33d623fea725339ee4a1ecdda5dea57150df00f8"
Oct 29 23:00:21.801217 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.799104    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"]
Oct 29 23:00:21.801217 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.801102    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-metrics-certs\") pod \"maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:21.801217 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.801186    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-maas-api-tls\") pod \"maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:21.801682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.801238    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1133119f-4582-4d64-9036-b4ebdae3d993-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:21.801682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.801443    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck99g\" (UniqueName: \"kubernetes.io/projected/1133119f-4582-4d64-9036-b4ebdae3d993-kube-api-access-ck99g\") pod \"maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:21.837356 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.833229    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:21.880072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.880005    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc" podStartSLOduration=1.879984425 podStartE2EDuration="1.879984425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:21.875879565 +0000 UTC m=+2763.900655730" watchObservedRunningTime="2026-10-29 23:00:21.879984425 +0000 UTC m=+2763.904760557"
Oct 29 23:00:21.880395 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.880357    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"]
Oct 29 23:00:21.905200 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.903579    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck99g\" (UniqueName: \"kubernetes.io/projected/1133119f-4582-4d64-9036-b4ebdae3d993-kube-api-access-ck99g\") pod \"maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:21.905200 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.903855    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-metrics-certs\") pod \"maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:21.905200 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.904166    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-maas-api-tls\") pod \"maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:21.906944 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.906894    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1133119f-4582-4d64-9036-b4ebdae3d993-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:21.907908 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.907861    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1133119f-4582-4d64-9036-b4ebdae3d993-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:21.914163 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.914108    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-metrics-certs\") pod \"maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:21.917368 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.917302    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-maas-api-tls\") pod \"maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:21.929917 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:21.929890    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck99g\" (UniqueName: \"kubernetes.io/projected/1133119f-4582-4d64-9036-b4ebdae3d993-kube-api-access-ck99g\") pod \"maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:22.088925 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.047889    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:22.093731 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.092921    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:22.167171 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.167139    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b239203-2773-4ff8-bc28-604940a34500" path="/var/lib/kubelet/pods/0b239203-2773-4ff8-bc28-604940a34500/volumes"
Oct 29 23:00:22.170154 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.170129    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2247bdec-f506-4144-a05a-884528c5c132" path="/var/lib/kubelet/pods/2247bdec-f506-4144-a05a-884528c5c132/volumes"
Oct 29 23:00:22.173130 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.173099    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" path="/var/lib/kubelet/pods/2f461145-f50b-4ebd-b9fa-25edb7d2af1a/volumes"
Oct 29 23:00:22.173784 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.173764    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32dbf81d-0294-4922-b785-1a0b3950017a" path="/var/lib/kubelet/pods/32dbf81d-0294-4922-b785-1a0b3950017a/volumes"
Oct 29 23:00:22.177930 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.177909    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" path="/var/lib/kubelet/pods/e27bd901-d6a1-48c3-91fe-b9c12b3d7b76/volumes"
Oct 29 23:00:22.178596 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.178576    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"]
Oct 29 23:00:22.178694 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.178614    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-22d634-29888580-w8bds"]
Oct 29 23:00:22.180994 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.180708    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"
Oct 29 23:00:22.234347 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.233435    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:22.284919 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.284534    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"
Oct 29 23:00:22.286835 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.286762    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"
Oct 29 23:00:22.305167 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.305088    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"
Oct 29 23:00:22.329397 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.329368    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-kube-api-access-tvzpb\" (UniqueName: \"kubernetes.io/projected/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-kube-api-access-tvzpb\") pod \"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c\" (UID: \"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c\") "
Oct 29 23:00:22.329636 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.329465    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-maas-api-service-ca\") pod \"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c\" (UID: \"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c\") "
Oct 29 23:00:22.329910 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.329882    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-maas-api-service-ca" pod "4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" (UID: "4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:22.330238 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.330223    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.332132 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.332101    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-kube-api-access-tvzpb" pod "4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" (UID: "4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c"). InnerVolumeSpecName "kube-api-access-tvzpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:22.431612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.431579    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521dba0b-944b-4ee0-8e00-60fa687c94bd-kube-api-access-xt6m8\" (UniqueName: \"kubernetes.io/projected/521dba0b-944b-4ee0-8e00-60fa687c94bd-kube-api-access-xt6m8\") pod \"521dba0b-944b-4ee0-8e00-60fa687c94bd\" (UID: \"521dba0b-944b-4ee0-8e00-60fa687c94bd\") "
Oct 29 23:00:22.431873 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.431851    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7a942f-a12b-42b1-8b11-b6e2c5103798-kube-api-access-g2v7s\" (UniqueName: \"kubernetes.io/projected/7c7a942f-a12b-42b1-8b11-b6e2c5103798-kube-api-access-g2v7s\") pod \"7c7a942f-a12b-42b1-8b11-b6e2c5103798\" (UID: \"7c7a942f-a12b-42b1-8b11-b6e2c5103798\") "
Oct 29 23:00:22.432160 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.432047    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b151b922-7435-4d4e-a403-3c911de5d8f6-kube-api-access-whxmd\" (UniqueName: \"kubernetes.io/projected/b151b922-7435-4d4e-a403-3c911de5d8f6-kube-api-access-whxmd\") pod \"b151b922-7435-4d4e-a403-3c911de5d8f6\" (UID: \"b151b922-7435-4d4e-a403-3c911de5d8f6\") "
Oct 29 23:00:22.432406 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.432161    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521dba0b-944b-4ee0-8e00-60fa687c94bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/521dba0b-944b-4ee0-8e00-60fa687c94bd-maas-api-service-ca\") pod \"521dba0b-944b-4ee0-8e00-60fa687c94bd\" (UID: \"521dba0b-944b-4ee0-8e00-60fa687c94bd\") "
Oct 29 23:00:22.432406 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.432302    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7a942f-a12b-42b1-8b11-b6e2c5103798-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a942f-a12b-42b1-8b11-b6e2c5103798-maas-api-service-ca\") pod \"7c7a942f-a12b-42b1-8b11-b6e2c5103798\" (UID: \"7c7a942f-a12b-42b1-8b11-b6e2c5103798\") "
Oct 29 23:00:22.432787 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.432500    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b151b922-7435-4d4e-a403-3c911de5d8f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b151b922-7435-4d4e-a403-3c911de5d8f6-maas-api-service-ca\") pod \"b151b922-7435-4d4e-a403-3c911de5d8f6\" (UID: \"b151b922-7435-4d4e-a403-3c911de5d8f6\") "
Oct 29 23:00:22.433374 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.433300    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvzpb\" (UniqueName: \"kubernetes.io/projected/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c-kube-api-access-tvzpb\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.434539 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.433979    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b151b922-7435-4d4e-a403-3c911de5d8f6-maas-api-service-ca" pod "b151b922-7435-4d4e-a403-3c911de5d8f6" (UID: "b151b922-7435-4d4e-a403-3c911de5d8f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:22.434539 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.434398    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7a942f-a12b-42b1-8b11-b6e2c5103798-maas-api-service-ca" pod "7c7a942f-a12b-42b1-8b11-b6e2c5103798" (UID: "7c7a942f-a12b-42b1-8b11-b6e2c5103798"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:22.434539 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.434491    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521dba0b-944b-4ee0-8e00-60fa687c94bd-maas-api-service-ca" pod "521dba0b-944b-4ee0-8e00-60fa687c94bd" (UID: "521dba0b-944b-4ee0-8e00-60fa687c94bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:22.442525 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.442488    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7a942f-a12b-42b1-8b11-b6e2c5103798-kube-api-access-g2v7s" pod "7c7a942f-a12b-42b1-8b11-b6e2c5103798" (UID: "7c7a942f-a12b-42b1-8b11-b6e2c5103798"). InnerVolumeSpecName "kube-api-access-g2v7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:22.450703 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.450667    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b151b922-7435-4d4e-a403-3c911de5d8f6-kube-api-access-whxmd" pod "b151b922-7435-4d4e-a403-3c911de5d8f6" (UID: "b151b922-7435-4d4e-a403-3c911de5d8f6"). InnerVolumeSpecName "kube-api-access-whxmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:22.452415 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.452381    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521dba0b-944b-4ee0-8e00-60fa687c94bd-kube-api-access-xt6m8" pod "521dba0b-944b-4ee0-8e00-60fa687c94bd" (UID: "521dba0b-944b-4ee0-8e00-60fa687c94bd"). InnerVolumeSpecName "kube-api-access-xt6m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:22.518609 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:00:22.518567    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1133119f_4582_4d64_9036_b4ebdae3d993.slice/crio-5b46e7f1f98b0ce71ee661b7455968959dcd4dac7edfcf4d03b494e6441d00a8 WatchSource:0}: Error finding container 5b46e7f1f98b0ce71ee661b7455968959dcd4dac7edfcf4d03b494e6441d00a8: Status 404 returned error can't find the container with id 5b46e7f1f98b0ce71ee661b7455968959dcd4dac7edfcf4d03b494e6441d00a8
Oct 29 23:00:22.530849 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.530819    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"]
Oct 29 23:00:22.535743 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.535720    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/521dba0b-944b-4ee0-8e00-60fa687c94bd-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.535861 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.535845    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a942f-a12b-42b1-8b11-b6e2c5103798-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.535942 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.535865    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b151b922-7435-4d4e-a403-3c911de5d8f6-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.535942 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.535878    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt6m8\" (UniqueName: \"kubernetes.io/projected/521dba0b-944b-4ee0-8e00-60fa687c94bd-kube-api-access-xt6m8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.535942 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.535892    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2v7s\" (UniqueName: \"kubernetes.io/projected/7c7a942f-a12b-42b1-8b11-b6e2c5103798-kube-api-access-g2v7s\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.535942 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.535905    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whxmd\" (UniqueName: \"kubernetes.io/projected/b151b922-7435-4d4e-a403-3c911de5d8f6-kube-api-access-whxmd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.777603 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.777515    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks" event={"ID":"4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c","Type":"ContainerDied","Data":"7f92267be92b6b367d8aac7098db31682674b6c0caf0df7a1e2f97269adbf1a3"}
Oct 29 23:00:22.777603 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.777584    4860 scope.go:122] "RemoveContainer" containerID="a07dafaa18b9a8b73e43c53f8b02fab67bb59016338a76f3cfc7c6c7161353bb"
Oct 29 23:00:22.778116 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.777524    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"
Oct 29 23:00:22.779431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.779407    4860 generic.go:356] "Generic (PLEG): container finished" podID="ec6cc782-8039-4b69-af59-053bc06853ce" containerID="55076b5bdb1943b44c97f28727e06b247d4b05d896957c94d637493c36f0b69f" exitCode=0
Oct 29 23:00:22.779536 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.779485    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc" event={"ID":"ec6cc782-8039-4b69-af59-053bc06853ce","Type":"ContainerDied","Data":"55076b5bdb1943b44c97f28727e06b247d4b05d896957c94d637493c36f0b69f"}
Oct 29 23:00:22.781040 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.781021    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"
Oct 29 23:00:22.781160 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.781024    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n" event={"ID":"7c7a942f-a12b-42b1-8b11-b6e2c5103798","Type":"ContainerDied","Data":"d2e1b5f6de4eccd9dce83a527314a45d8b41e70290dd0f04041128366ea20748"}
Oct 29 23:00:22.782730 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.782704    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl" event={"ID":"521dba0b-944b-4ee0-8e00-60fa687c94bd","Type":"ContainerDied","Data":"e11cacb6731588fbfd847f22aa2c961fccb4628213b1ccda5f1c023ca67b7934"}
Oct 29 23:00:22.782830 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.782750    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"
Oct 29 23:00:22.793155 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.793130    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x" event={"ID":"1133119f-4582-4d64-9036-b4ebdae3d993","Type":"ContainerStarted","Data":"5b46e7f1f98b0ce71ee661b7455968959dcd4dac7edfcf4d03b494e6441d00a8"}
Oct 29 23:00:22.795649 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.795630    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"
Oct 29 23:00:22.795738 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.795661    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k" event={"ID":"b151b922-7435-4d4e-a403-3c911de5d8f6","Type":"ContainerDied","Data":"3ccced3def1396db9e64e3cb882a8255842a24d1af9959a9b0f12bb6b9dcb327"}
Oct 29 23:00:22.799166 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.799143    4860 scope.go:122] "RemoveContainer" containerID="b9952b3c710081a74381d8b6622aece60726a752d31fa5b8751715540dc5a269"
Oct 29 23:00:22.812095 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.812073    4860 scope.go:122] "RemoveContainer" containerID="305277bf5f0c396ebe6a9c5a0f0d82143e20c0aaefeb8968df7ea7973b0bc809"
Oct 29 23:00:22.830473 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.830445    4860 scope.go:122] "RemoveContainer" containerID="d3f808c60e611f12403a7acc16e9e0db409e9ee954a10b680ec0a4fa9767b781"
Oct 29 23:00:22.903643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.903145    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"]
Oct 29 23:00:22.912720 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.912689    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-80248b-29888580-nv5ks"]
Oct 29 23:00:22.982344 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.976456    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"]
Oct 29 23:00:22.987404 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:22.984354    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-784e7a-29888580-w6rzl"]
Oct 29 23:00:23.005388 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.005341    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"
Oct 29 23:00:23.150405 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.143803    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"]
Oct 29 23:00:23.172350 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.172171    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-kube-api-access-jfv7v\" (UniqueName: \"kubernetes.io/projected/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-kube-api-access-jfv7v\") pod \"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d\" (UID: \"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d\") "
Oct 29 23:00:23.172528 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.172444    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-maas-api-service-ca\") pod \"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d\" (UID: \"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d\") "
Oct 29 23:00:23.177933 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.177905    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-kube-api-access-jfv7v" pod "bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" (UID: "bf8314b5-c567-4ebd-8a0b-d55cf6b3047d"). InnerVolumeSpecName "kube-api-access-jfv7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:23.188578 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.188538    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-maas-api-service-ca" pod "bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" (UID: "bf8314b5-c567-4ebd-8a0b-d55cf6b3047d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:23.223159 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.223124    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-9aafe6-29888580-bw85n"]
Oct 29 23:00:23.260462 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.260432    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"]
Oct 29 23:00:23.266552 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.265714    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-85862e-29888580-95c4k"]
Oct 29 23:00:23.274594 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.274567    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-maas-api-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:23.274785 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.274770    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfv7v\" (UniqueName: \"kubernetes.io/projected/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d-kube-api-access-jfv7v\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:23.817204 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.817162    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4" event={"ID":"bf8314b5-c567-4ebd-8a0b-d55cf6b3047d","Type":"ContainerDied","Data":"e9bf2bcb78373c0c5c41c3a9f375fc8875f327d393e83228eedf07c2d9cadfd6"}
Oct 29 23:00:23.817744 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.817240    4860 scope.go:122] "RemoveContainer" containerID="88626d8725a43348b53069880d111b92d1ef2fe23f11c91cd711bc6c19b448b9"
Oct 29 23:00:23.817744 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.817365    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"
Oct 29 23:00:23.826114 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.825452    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x" event={"ID":"1133119f-4582-4d64-9036-b4ebdae3d993","Type":"ContainerStarted","Data":"3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0"}
Oct 29 23:00:23.826773 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.826414    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:23.850150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.850120    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:23.912759 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.910940    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"]
Oct 29 23:00:23.931502 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.930246    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-eb02df-29888580-zmxh4"]
Oct 29 23:00:23.936356 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:23.936262    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x" podStartSLOduration=2.611922446 podStartE2EDuration="2.936241291s" totalImagesPullingTime="324.318845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:23.930644198 +0000 UTC m=+2765.955420329" watchObservedRunningTime="2026-10-29 23:00:23.936241291 +0000 UTC m=+2765.961017423"
Oct 29 23:00:24.176855 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.176816    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05" path="/var/lib/kubelet/pods/4e276a4c-e944-495d-8d4e-f80ebd436f05/volumes"
Oct 29 23:00:24.177580 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.177550    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" path="/var/lib/kubelet/pods/4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c/volumes"
Oct 29 23:00:24.178203 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.178178    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd" path="/var/lib/kubelet/pods/521dba0b-944b-4ee0-8e00-60fa687c94bd/volumes"
Oct 29 23:00:24.178203 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.178193    4860 scope.go:122] "RemoveContainer" containerID="98949dbac696c2dfd25919aca39877b9292195525ea6ca35729987c95dc0d02e"
Oct 29 23:00:24.178476 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.178455    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:24.179305 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.179280    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" path="/var/lib/kubelet/pods/7c7a942f-a12b-42b1-8b11-b6e2c5103798/volumes"
Oct 29 23:00:24.180869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.180699    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6" path="/var/lib/kubelet/pods/b151b922-7435-4d4e-a403-3c911de5d8f6/volumes"
Oct 29 23:00:24.181536 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.181303    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" path="/var/lib/kubelet/pods/bf8314b5-c567-4ebd-8a0b-d55cf6b3047d/volumes"
Oct 29 23:00:24.200726 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.200701    4860 scope.go:122] "RemoveContainer" containerID="4aeb738f503c7209ad2c06d76a63597ecc5b2423508e2dc44a5e414e5c5bde66"
Oct 29 23:00:24.232645 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.232607    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-openshift-ca\") pod \"ec6cc782-8039-4b69-af59-053bc06853ce\" (UID: \"ec6cc782-8039-4b69-af59-053bc06853ce\") "
Oct 29 23:00:24.232774 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.232719    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-xks-ca\") pod \"ec6cc782-8039-4b69-af59-053bc06853ce\" (UID: \"ec6cc782-8039-4b69-af59-053bc06853ce\") "
Oct 29 23:00:24.232774 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.232757    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6cc782-8039-4b69-af59-053bc06853ce-kube-api-access-zj8jj\" (UniqueName: \"kubernetes.io/projected/ec6cc782-8039-4b69-af59-053bc06853ce-kube-api-access-zj8jj\") pod \"ec6cc782-8039-4b69-af59-053bc06853ce\" (UID: \"ec6cc782-8039-4b69-af59-053bc06853ce\") "
Oct 29 23:00:24.233883 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.233840    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-openshift-ca" pod "ec6cc782-8039-4b69-af59-053bc06853ce" (UID: "ec6cc782-8039-4b69-af59-053bc06853ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:24.236587 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.236560    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-xks-ca" pod "ec6cc782-8039-4b69-af59-053bc06853ce" (UID: "ec6cc782-8039-4b69-af59-053bc06853ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:24.252645 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.252621    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6cc782-8039-4b69-af59-053bc06853ce-kube-api-access-zj8jj" pod "ec6cc782-8039-4b69-af59-053bc06853ce" (UID: "ec6cc782-8039-4b69-af59-053bc06853ce"). InnerVolumeSpecName "kube-api-access-zj8jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:24.335166 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.335136    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.335287 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.335170    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6cc782-8039-4b69-af59-053bc06853ce-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.335287 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.335183    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj8jj\" (UniqueName: \"kubernetes.io/projected/ec6cc782-8039-4b69-af59-053bc06853ce-kube-api-access-zj8jj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:24.938065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.938033    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"
Oct 29 23:00:24.942444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.942404    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc" event={"ID":"ec6cc782-8039-4b69-af59-053bc06853ce","Type":"ContainerDied","Data":"bbb929646f60be6927b2db3bc4b77f8bc0333a39bc95c7381e7170a53e41269f"}
Oct 29 23:00:24.942566 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:24.942547    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbb929646f60be6927b2db3bc4b77f8bc0333a39bc95c7381e7170a53e41269f"
Oct 29 23:00:25.177636 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.177600    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 23:00:25.367706 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.367618    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-credential-socket\") pod \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") "
Oct 29 23:00:25.367869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.367812    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-podinfo\") pod \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") "
Oct 29 23:00:25.367869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.367850    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-certs\") pod \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") "
Oct 29 23:00:25.368000 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.367894    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-envoy\") pod \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") "
Oct 29 23:00:25.368000 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.367891    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-credential-socket" pod "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" (UID: "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:25.368000 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.367956    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istiod-ca-cert\") pod \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") "
Oct 29 23:00:25.368159 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.368007    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-kube-api-access-5xnhs\" (UniqueName: \"kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-kube-api-access-5xnhs\") pod \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") "
Oct 29 23:00:25.368159 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.368046    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-token\" (UniqueName: \"kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-token\") pod \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") "
Oct 29 23:00:25.368159 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.368142    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-socket\") pod \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") "
Oct 29 23:00:25.368366 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.368248    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-data\") pod \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\" (UID: \"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c\") "
Oct 29 23:00:25.368584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.368243    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-certs" pod "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" (UID: "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:25.368584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.368467    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-socket" pod "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" (UID: "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:25.368729 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.368626    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-data" pod "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" (UID: "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:25.368787 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.368739    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istiod-ca-cert" pod "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" (UID: "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:25.369249 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.369225    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.369450 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.369431    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.369565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.369551    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.369677 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.369662    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.369771 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.369759    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.372369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.370965    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-podinfo" pod "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" (UID: "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:25.372369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.371044    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-envoy" pod "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" (UID: "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:25.372369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.371171    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-token" pod "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" (UID: "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:25.372369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.371462    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-kube-api-access-5xnhs" pod "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" (UID: "55e501b2-4c2a-4271-a9f5-5435e6ea3a9c"). InnerVolumeSpecName "kube-api-access-5xnhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:25.483346 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.472068    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.483346 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.472104    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.483346 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.472119    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xnhs\" (UniqueName: \"kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-kube-api-access-5xnhs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.483346 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.472130    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:25.967873 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.967776    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:25.976310 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.976276    4860 generic.go:356] "Generic (PLEG): container finished" podID="55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" containerID="ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02" exitCode=0
Oct 29 23:00:25.976548 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.976353    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"
Oct 29 23:00:25.976548 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.976357    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6" event={"ID":"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c","Type":"ContainerDied","Data":"ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02"}
Oct 29 23:00:25.976548 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.976476    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6" event={"ID":"55e501b2-4c2a-4271-a9f5-5435e6ea3a9c","Type":"ContainerDied","Data":"c4b97328150e21ed20daf5ad5196aa0337b6318d9a7646bf9fa3cff380b95f58"}
Oct 29 23:00:25.976548 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.976505    4860 scope.go:122] "RemoveContainer" containerID="ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02"
Oct 29 23:00:25.990191 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.990171    4860 scope.go:122] "RemoveContainer" containerID="ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02"
Oct 29 23:00:25.990511 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:25.990488    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02\": container with ID starting with ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02 not found: ID does not exist" containerID="ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02"
Oct 29 23:00:25.990566 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:25.990520    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02"} err="failed to get container status \"ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02\": rpc error: code = NotFound desc = could not find container \"ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02\": container with ID starting with ae615cc468d1fcc0b39908c2ddd4d839dbfdf292dac53039934fa140a13d6d02 not found: ID does not exist"
Oct 29 23:00:26.038606 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:26.038578    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"]
Oct 29 23:00:26.052715 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:26.052690    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6"]
Oct 29 23:00:26.098666 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:26.098633    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:26.118970 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:26.118942    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" path="/var/lib/kubelet/pods/55e501b2-4c2a-4271-a9f5-5435e6ea3a9c/volumes"
Oct 29 23:00:26.141954 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:26.141931    4860 patch_prober.go:29] interesting pod/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:00:26.142071 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:26.141985    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-aaa3c6-openshift-default-7cf94db6f9-x9gf6" podUID="55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:27.843483 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:27.843452    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:29.409813 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:29.409778    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:29.947835 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:29.947804    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:00:30.668198 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:30.668166    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"]
Oct 29 23:00:30.703100 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:30.703073    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"]
Oct 29 23:00:30.853156 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:30.853115    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:30.983612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:30.983547    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:31.874457 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:31.874424    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:33.127908 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:33.127878    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb_f91089d4-1a38-4d94-87f8-8131ab078130/payload-processing/0.log"
Oct 29 23:00:33.303538 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:33.303506    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:33.325999 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:33.325977    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-8wn8h_23a26fe4-1010-464f-9df0-ff85860cc488/payload-processing/0.log"
Oct 29 23:00:33.518144 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:33.518113    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-8wn8h_23a26fe4-1010-464f-9df0-ff85860cc488/payload-processing/0.log"
Oct 29 23:00:33.868848 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:33.868762    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:33.973534 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:33.973502    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"]
Oct 29 23:00:33.986857 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:33.986822    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-3bd0f6-pt6cc"]
Oct 29 23:00:33.988789 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:33.988762    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:34.122191 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:34.122074    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:34.130682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:34.130650    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec6cc782-8039-4b69-af59-053bc06853ce" path="/var/lib/kubelet/pods/ec6cc782-8039-4b69-af59-053bc06853ce/volumes"
Oct 29 23:00:34.476254 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:34.476220    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:34.771005 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:34.770906    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:34.896766 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:34.896723    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:35.030522 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:35.030425    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:35.668710 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:35.668669    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t" podUID="c7208a84-7581-40bd-8b3e-f8d399feabb0" containerName="payload-processing" containerID="cri-o://bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654" gracePeriod=25
Oct 29 23:00:35.703778 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:35.703740    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6" podUID="335d830d-7cdd-4527-a215-6f247fbf11b8" containerName="payload-pre-processing" containerID="cri-o://b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a" gracePeriod=25
Oct 29 23:00:35.746016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:35.745986    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:37.592122 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:37.592083    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:39.443779 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:39.443746    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:40.845540 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:40.844917    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:40.974086 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:40.974056    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:41.082955 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:41.082921    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"]
Oct 29 23:00:41.083444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:41.083401    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k" podUID="9e56389b-3fc6-4877-9034-efdf79f2df59" containerName="istio-proxy" containerID="cri-o://32f9a080bb6d1a07d14a80c3a751490486cfb7bf71149e1a830fc9ba7ce5f41f" gracePeriod=30
Oct 29 23:00:41.139827 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:41.139742    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:41.295347 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:41.295147    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:41.576359 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:41.576278    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:42.855502 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:42.855468    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:43.736765 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:43.736724    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t" podUID="c7208a84-7581-40bd-8b3e-f8d399feabb0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:44.575934 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:44.575877    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:44.745007 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:44.744969    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6" podUID="335d830d-7cdd-4527-a215-6f247fbf11b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:46.302727 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.302416    4860 generic.go:356] "Generic (PLEG): container finished" podID="9e56389b-3fc6-4877-9034-efdf79f2df59" containerID="32f9a080bb6d1a07d14a80c3a751490486cfb7bf71149e1a830fc9ba7ce5f41f" exitCode=0
Oct 29 23:00:46.302727 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.302632    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k" event={"ID":"9e56389b-3fc6-4877-9034-efdf79f2df59","Type":"ContainerDied","Data":"32f9a080bb6d1a07d14a80c3a751490486cfb7bf71149e1a830fc9ba7ce5f41f"}
Oct 29 23:00:46.369632 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.369573    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 23:00:46.445445 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.445408    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-kube-api-access-j99br\" (UniqueName: \"kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-kube-api-access-j99br\") pod \"9e56389b-3fc6-4877-9034-efdf79f2df59\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") "
Oct 29 23:00:46.445630 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.445496    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-token\" (UniqueName: \"kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-token\") pod \"9e56389b-3fc6-4877-9034-efdf79f2df59\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") "
Oct 29 23:00:46.445630 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.445588    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-certs\") pod \"9e56389b-3fc6-4877-9034-efdf79f2df59\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") "
Oct 29 23:00:46.445630 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.445618    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-envoy\") pod \"9e56389b-3fc6-4877-9034-efdf79f2df59\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") "
Oct 29 23:00:46.445816 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.445790    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e56389b-3fc6-4877-9034-efdf79f2df59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e56389b-3fc6-4877-9034-efdf79f2df59-istiod-ca-cert\") pod \"9e56389b-3fc6-4877-9034-efdf79f2df59\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") "
Oct 29 23:00:46.445886 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.445823    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-socket\") pod \"9e56389b-3fc6-4877-9034-efdf79f2df59\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") "
Oct 29 23:00:46.445948 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.445914    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-certs" pod "9e56389b-3fc6-4877-9034-efdf79f2df59" (UID: "9e56389b-3fc6-4877-9034-efdf79f2df59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.446030 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.446008    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-podinfo\") pod \"9e56389b-3fc6-4877-9034-efdf79f2df59\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") "
Oct 29 23:00:46.446093 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.446065    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-credential-socket\") pod \"9e56389b-3fc6-4877-9034-efdf79f2df59\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") "
Oct 29 23:00:46.446145 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.446097    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-data\") pod \"9e56389b-3fc6-4877-9034-efdf79f2df59\" (UID: \"9e56389b-3fc6-4877-9034-efdf79f2df59\") "
Oct 29 23:00:46.446626 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.446245    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e56389b-3fc6-4877-9034-efdf79f2df59-istiod-ca-cert" pod "9e56389b-3fc6-4877-9034-efdf79f2df59" (UID: "9e56389b-3fc6-4877-9034-efdf79f2df59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:46.446626 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.446485    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-socket" pod "9e56389b-3fc6-4877-9034-efdf79f2df59" (UID: "9e56389b-3fc6-4877-9034-efdf79f2df59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.446626 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.446587    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-credential-socket" pod "9e56389b-3fc6-4877-9034-efdf79f2df59" (UID: "9e56389b-3fc6-4877-9034-efdf79f2df59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.446901 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.446730    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-data" pod "9e56389b-3fc6-4877-9034-efdf79f2df59" (UID: "9e56389b-3fc6-4877-9034-efdf79f2df59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.446993 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.446967    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e56389b-3fc6-4877-9034-efdf79f2df59-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.447064 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.446995    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.447064 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.447009    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.447064 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.447023    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.447064 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.447035    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.449208 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.449179    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-token" pod "9e56389b-3fc6-4877-9034-efdf79f2df59" (UID: "9e56389b-3fc6-4877-9034-efdf79f2df59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:46.449672 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.449646    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-kube-api-access-j99br" pod "9e56389b-3fc6-4877-9034-efdf79f2df59" (UID: "9e56389b-3fc6-4877-9034-efdf79f2df59"). InnerVolumeSpecName "kube-api-access-j99br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:46.449999 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.449974    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-podinfo" pod "9e56389b-3fc6-4877-9034-efdf79f2df59" (UID: "9e56389b-3fc6-4877-9034-efdf79f2df59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:46.450084 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.450030    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-envoy" pod "9e56389b-3fc6-4877-9034-efdf79f2df59" (UID: "9e56389b-3fc6-4877-9034-efdf79f2df59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.523663 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.523628    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"]
Oct 29 23:00:46.545502 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.545474    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"]
Oct 29 23:00:46.549034 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.549009    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.549034 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.549035    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.549189 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.549045    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j99br\" (UniqueName: \"kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-kube-api-access-j99br\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.549189 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.549056    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e56389b-3fc6-4877-9034-efdf79f2df59-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.645636 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.645570    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:46.779143 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:46.779105    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:47.313451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:47.313421    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"
Oct 29 23:00:47.313922 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:47.313461    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k" event={"ID":"9e56389b-3fc6-4877-9034-efdf79f2df59","Type":"ContainerDied","Data":"ea5e293a09bff846c206e9167e258062cd5255cba58e2e0ddad5ef5e1bcb8e59"}
Oct 29 23:00:47.313922 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:47.313534    4860 scope.go:122] "RemoveContainer" containerID="32f9a080bb6d1a07d14a80c3a751490486cfb7bf71149e1a830fc9ba7ce5f41f"
Oct 29 23:00:47.373384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:47.373347    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"]
Oct 29 23:00:47.386712 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:47.386686    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3bd0f6-openshift-default-69c4758d77-74t4k"]
Oct 29 23:00:48.057652 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:48.057618    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:00:48.116742 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:48.116709    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e56389b-3fc6-4877-9034-efdf79f2df59" path="/var/lib/kubelet/pods/9e56389b-3fc6-4877-9034-efdf79f2df59/volumes"
Oct 29 23:00:48.833295 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:48.833261    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:50.356362 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:50.356308    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:51.113681 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:51.113645    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:51.237804 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:51.237767    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:51.354560 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:51.354519    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:51.731035 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:51.730994    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:52.085254 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:52.085158    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"]
Oct 29 23:00:52.131111 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:52.131077    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"]
Oct 29 23:00:52.252826 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:52.252788    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:52.424317 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:52.424209    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:52.569556 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:52.569517    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:53.170382 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:53.170340    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:53.336061 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:53.336023    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:53.496740 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:53.496669    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:53.753412 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:53.751625    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t" podUID="c7208a84-7581-40bd-8b3e-f8d399feabb0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:54.745104 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:54.745060    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6" podUID="335d830d-7cdd-4527-a215-6f247fbf11b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:55.132605 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:55.132508    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:56.050034 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.050000    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"
Oct 29 23:00:56.129521 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.129493    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7208a84-7581-40bd-8b3e-f8d399feabb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7208a84-7581-40bd-8b3e-f8d399feabb0-plugins-config-volume\") pod \"c7208a84-7581-40bd-8b3e-f8d399feabb0\" (UID: \"c7208a84-7581-40bd-8b3e-f8d399feabb0\") "
Oct 29 23:00:56.129676 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.129558    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7208a84-7581-40bd-8b3e-f8d399feabb0-kube-api-access-6vmtk\" (UniqueName: \"kubernetes.io/projected/c7208a84-7581-40bd-8b3e-f8d399feabb0-kube-api-access-6vmtk\") pod \"c7208a84-7581-40bd-8b3e-f8d399feabb0\" (UID: \"c7208a84-7581-40bd-8b3e-f8d399feabb0\") "
Oct 29 23:00:56.131628 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.130792    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7208a84-7581-40bd-8b3e-f8d399feabb0-plugins-config-volume" pod "c7208a84-7581-40bd-8b3e-f8d399feabb0" (UID: "c7208a84-7581-40bd-8b3e-f8d399feabb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:56.133252 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.133222    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7208a84-7581-40bd-8b3e-f8d399feabb0-kube-api-access-6vmtk" pod "c7208a84-7581-40bd-8b3e-f8d399feabb0" (UID: "c7208a84-7581-40bd-8b3e-f8d399feabb0"). InnerVolumeSpecName "kube-api-access-6vmtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:56.157186 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.157159    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"
Oct 29 23:00:56.231688 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.231609    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335d830d-7cdd-4527-a215-6f247fbf11b8-kube-api-access-mdkpj\" (UniqueName: \"kubernetes.io/projected/335d830d-7cdd-4527-a215-6f247fbf11b8-kube-api-access-mdkpj\") pod \"335d830d-7cdd-4527-a215-6f247fbf11b8\" (UID: \"335d830d-7cdd-4527-a215-6f247fbf11b8\") "
Oct 29 23:00:56.231865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.231819    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335d830d-7cdd-4527-a215-6f247fbf11b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335d830d-7cdd-4527-a215-6f247fbf11b8-plugins-config-volume\") pod \"335d830d-7cdd-4527-a215-6f247fbf11b8\" (UID: \"335d830d-7cdd-4527-a215-6f247fbf11b8\") "
Oct 29 23:00:56.232250 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.232221    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335d830d-7cdd-4527-a215-6f247fbf11b8-plugins-config-volume" pod "335d830d-7cdd-4527-a215-6f247fbf11b8" (UID: "335d830d-7cdd-4527-a215-6f247fbf11b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:56.232474 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.232456    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7208a84-7581-40bd-8b3e-f8d399feabb0-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:56.232567 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.232475    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vmtk\" (UniqueName: \"kubernetes.io/projected/c7208a84-7581-40bd-8b3e-f8d399feabb0-kube-api-access-6vmtk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:56.232567 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.232484    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335d830d-7cdd-4527-a215-6f247fbf11b8-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:56.233874 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.233841    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335d830d-7cdd-4527-a215-6f247fbf11b8-kube-api-access-mdkpj" pod "335d830d-7cdd-4527-a215-6f247fbf11b8" (UID: "335d830d-7cdd-4527-a215-6f247fbf11b8"). InnerVolumeSpecName "kube-api-access-mdkpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:56.334128 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.334094    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdkpj\" (UniqueName: \"kubernetes.io/projected/335d830d-7cdd-4527-a215-6f247fbf11b8-kube-api-access-mdkpj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:56.420785 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.420747    4860 generic.go:356] "Generic (PLEG): container finished" podID="335d830d-7cdd-4527-a215-6f247fbf11b8" containerID="b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a" exitCode=0
Oct 29 23:00:56.420965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.420794    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"
Oct 29 23:00:56.420965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.420835    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6" event={"ID":"335d830d-7cdd-4527-a215-6f247fbf11b8","Type":"ContainerDied","Data":"b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a"}
Oct 29 23:00:56.420965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.420889    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6" event={"ID":"335d830d-7cdd-4527-a215-6f247fbf11b8","Type":"ContainerDied","Data":"ff7e7ba7f820abf0991c8554517c936939e16c5a47bc0f9a0cb2b1e4d898fa16"}
Oct 29 23:00:56.420965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.420919    4860 scope.go:122] "RemoveContainer" containerID="b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a"
Oct 29 23:00:56.422653 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.422625    4860 generic.go:356] "Generic (PLEG): container finished" podID="c7208a84-7581-40bd-8b3e-f8d399feabb0" containerID="bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654" exitCode=0
Oct 29 23:00:56.422793 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.422676    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"
Oct 29 23:00:56.422793 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.422742    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t" event={"ID":"c7208a84-7581-40bd-8b3e-f8d399feabb0","Type":"ContainerDied","Data":"bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654"}
Oct 29 23:00:56.422947 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.422798    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t" event={"ID":"c7208a84-7581-40bd-8b3e-f8d399feabb0","Type":"ContainerDied","Data":"c43f65813ff09552c756fa08dc4ad8edeecf2b13aef837e422df3b5143f7706d"}
Oct 29 23:00:56.433829 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.433810    4860 scope.go:122] "RemoveContainer" containerID="b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a"
Oct 29 23:00:56.434082 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:56.434066    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a\": container with ID starting with b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a not found: ID does not exist" containerID="b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a"
Oct 29 23:00:56.434138 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.434089    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a"} err="failed to get container status \"b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a\": rpc error: code = NotFound desc = could not find container \"b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a\": container with ID starting with b38a1adb09b49e503a0971fa5ea73df857a0a38064375bc381941d6773ee8d4a not found: ID does not exist"
Oct 29 23:00:56.434138 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.434106    4860 scope.go:122] "RemoveContainer" containerID="bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654"
Oct 29 23:00:56.444797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.444779    4860 scope.go:122] "RemoveContainer" containerID="bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654"
Oct 29 23:00:56.445053 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:00:56.445034    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654\": container with ID starting with bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654 not found: ID does not exist" containerID="bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654"
Oct 29 23:00:56.445102 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.445058    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654"} err="failed to get container status \"bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654\": rpc error: code = NotFound desc = could not find container \"bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654\": container with ID starting with bb7d02b44cef8b377f0850c6ab4817ae451bee06b8856d47430775cf1b417654 not found: ID does not exist"
Oct 29 23:00:56.498386 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.498284    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"]
Oct 29 23:00:56.503015 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.502991    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3bd0f6-858c665c67-btrf6"]
Oct 29 23:00:56.532436 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.532410    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"]
Oct 29 23:00:56.536578 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.536556    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3bd0f6-c58c9678-lhc7t"]
Oct 29 23:00:56.917401 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:56.916939    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:57.091349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:57.091156    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc" podUID="056308d8-3cce-4df1-a0a9-e0486218e4f5" containerName="payload-processing" containerID="cri-o://dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa" gracePeriod=25
Oct 29 23:00:57.131555 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:57.131525    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw" podUID="f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43" containerName="payload-pre-processing" containerID="cri-o://f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0" gracePeriod=25
Oct 29 23:00:57.894072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:57.893997    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:58.019795 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.019761    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:58.120747 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.120708    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335d830d-7cdd-4527-a215-6f247fbf11b8" path="/var/lib/kubelet/pods/335d830d-7cdd-4527-a215-6f247fbf11b8/volumes"
Oct 29 23:00:58.121494 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.121428    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7208a84-7581-40bd-8b3e-f8d399feabb0" path="/var/lib/kubelet/pods/c7208a84-7581-40bd-8b3e-f8d399feabb0/volumes"
Oct 29 23:00:58.158225 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.158140    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"]
Oct 29 23:00:58.158459 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.158409    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy" containerID="cri-o://6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0" gracePeriod=30
Oct 29 23:00:58.208965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.208928    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"]
Oct 29 23:00:58.209471 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.209419    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb" podUID="1d25d5c6-0825-4c7e-9926-4ab0add384c4" containerName="maas-api" containerID="cri-o://35536d3ab530f49d5ede8fcb1342d2b77822c52630a68124566af062d55f066d" gracePeriod=30
Oct 29 23:00:58.219829 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.219802    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:58.346606 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.346574    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:58.811767 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.811279    4860 generic.go:356] "Generic (PLEG): container finished" podID="1d25d5c6-0825-4c7e-9926-4ab0add384c4" containerID="35536d3ab530f49d5ede8fcb1342d2b77822c52630a68124566af062d55f066d" exitCode=0
Oct 29 23:00:58.811767 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.811655    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb" event={"ID":"1d25d5c6-0825-4c7e-9926-4ab0add384c4","Type":"ContainerDied","Data":"35536d3ab530f49d5ede8fcb1342d2b77822c52630a68124566af062d55f066d"}
Oct 29 23:00:58.842824 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.842794    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:58.926344 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.924845    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 23:00:58.929562 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.929131    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d25d5c6-0825-4c7e-9926-4ab0add384c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d25d5c6-0825-4c7e-9926-4ab0add384c4-openshift-service-ca\") pod \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") "
Oct 29 23:00:58.929562 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.929287    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-metrics-certs\") pod \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") "
Oct 29 23:00:58.929562 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.929402    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-maas-api-tls\") pod \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") "
Oct 29 23:00:58.929866 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.929611    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d25d5c6-0825-4c7e-9926-4ab0add384c4-openshift-service-ca" pod "1d25d5c6-0825-4c7e-9926-4ab0add384c4" (UID: "1d25d5c6-0825-4c7e-9926-4ab0add384c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:58.929866 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.929651    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d25d5c6-0825-4c7e-9926-4ab0add384c4-kube-api-access-tfxf8\" (UniqueName: \"kubernetes.io/projected/1d25d5c6-0825-4c7e-9926-4ab0add384c4-kube-api-access-tfxf8\") pod \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\" (UID: \"1d25d5c6-0825-4c7e-9926-4ab0add384c4\") "
Oct 29 23:00:58.930542 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.930516    4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d25d5c6-0825-4c7e-9926-4ab0add384c4-openshift-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:58.932299 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.932258    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-metrics-certs" pod "1d25d5c6-0825-4c7e-9926-4ab0add384c4" (UID: "1d25d5c6-0825-4c7e-9926-4ab0add384c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:58.936010 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.935983    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d25d5c6-0825-4c7e-9926-4ab0add384c4-kube-api-access-tfxf8" pod "1d25d5c6-0825-4c7e-9926-4ab0add384c4" (UID: "1d25d5c6-0825-4c7e-9926-4ab0add384c4"). InnerVolumeSpecName "kube-api-access-tfxf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:58.936420 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.936400    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-maas-api-tls" pod "1d25d5c6-0825-4c7e-9926-4ab0add384c4" (UID: "1d25d5c6-0825-4c7e-9926-4ab0add384c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:58.965710 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:58.965622    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:59.031699 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:59.031661    4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-metrics-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:59.031699 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:59.031697    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d25d5c6-0825-4c7e-9926-4ab0add384c4-maas-api-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:59.032148 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:59.031710    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfxf8\" (UniqueName: \"kubernetes.io/projected/1d25d5c6-0825-4c7e-9926-4ab0add384c4-kube-api-access-tfxf8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:00:59.423708 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:59.423669    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:59.822542 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:59.822512    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"
Oct 29 23:00:59.822542 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:59.822525    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb" event={"ID":"1d25d5c6-0825-4c7e-9926-4ab0add384c4","Type":"ContainerDied","Data":"c71eea82a796446f84f8d377a3326c40bac253f9e72f547049b3a3dac2af2d90"}
Oct 29 23:00:59.822721 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:59.822591    4860 scope.go:122] "RemoveContainer" containerID="35536d3ab530f49d5ede8fcb1342d2b77822c52630a68124566af062d55f066d"
Oct 29 23:00:59.882162 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:59.882135    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"]
Oct 29 23:00:59.895639 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:00:59.895614    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c933cb-d9cdf56c9-jwqzb"]
Oct 29 23:01:00.119875 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:00.119800    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d25d5c6-0825-4c7e-9926-4ab0add384c4" path="/var/lib/kubelet/pods/1d25d5c6-0825-4c7e-9926-4ab0add384c4/volumes"
Oct 29 23:01:00.560775 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:00.560733    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:01.033971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.033940    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"]
Oct 29 23:01:01.048336 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.048290    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39f42f74-kr768"]
Oct 29 23:01:01.546207 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.546155    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main" containerID="cri-o://19a2ffaea059c86af3276569651c889b91a589a77ab2640a4ef02875b14d917f" gracePeriod=45
Oct 29 23:01:01.566752 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.566699    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main" containerID="cri-o://4a8dedc61c05bdbeb3d0547296b96802cab77c456df133b67650cceef06aa582" gracePeriod=45
Oct 29 23:01:01.886261 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.886233    4860 generic.go:356] "Generic (PLEG): container finished" podID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerID="19a2ffaea059c86af3276569651c889b91a589a77ab2640a4ef02875b14d917f" exitCode=0
Oct 29 23:01:01.886678 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.886632    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" event={"ID":"62e206a5-c7f6-4865-98bd-ca3a68f5312a","Type":"ContainerDied","Data":"19a2ffaea059c86af3276569651c889b91a589a77ab2640a4ef02875b14d917f"}
Oct 29 23:01:01.888602 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.888559    4860 generic.go:356] "Generic (PLEG): container finished" podID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerID="4a8dedc61c05bdbeb3d0547296b96802cab77c456df133b67650cceef06aa582" exitCode=0
Oct 29 23:01:01.888826 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.888808    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" event={"ID":"58e4fb66-49ec-436a-8e2a-8890f83ed081","Type":"ContainerDied","Data":"4a8dedc61c05bdbeb3d0547296b96802cab77c456df133b67650cceef06aa582"}
Oct 29 23:01:01.910734 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.910443    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 23:01:01.916742 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.916719    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 23:01:01.980406 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.980365    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e4fb66-49ec-436a-8e2a-8890f83ed081-kube-api-access-2pgqt\" (UniqueName: \"kubernetes.io/projected/58e4fb66-49ec-436a-8e2a-8890f83ed081-kube-api-access-2pgqt\") pod \"58e4fb66-49ec-436a-8e2a-8890f83ed081\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") "
Oct 29 23:01:01.980652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.980418    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-home\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-home\") pod \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") "
Oct 29 23:01:01.980652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.980464    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-model-cache\") pod \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") "
Oct 29 23:01:01.980652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.980563    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-model-cache\") pod \"58e4fb66-49ec-436a-8e2a-8890f83ed081\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") "
Oct 29 23:01:01.980652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.980614    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-home\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-home\") pod \"58e4fb66-49ec-436a-8e2a-8890f83ed081\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") "
Oct 29 23:01:01.980893 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.980698    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-tmp-dir\") pod \"58e4fb66-49ec-436a-8e2a-8890f83ed081\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") "
Oct 29 23:01:01.980893 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.980698    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-home" pod "62e206a5-c7f6-4865-98bd-ca3a68f5312a" (UID: "62e206a5-c7f6-4865-98bd-ca3a68f5312a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:01.980893 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.980769    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tmp-dir\") pod \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") "
Oct 29 23:01:01.980893 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.980820    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tls-certs\" (UniqueName: \"kubernetes.io/secret/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tls-certs\") pod \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") "
Oct 29 23:01:01.980893 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.980888    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e4fb66-49ec-436a-8e2a-8890f83ed081-tls-certs\" (UniqueName: \"kubernetes.io/secret/58e4fb66-49ec-436a-8e2a-8890f83ed081-tls-certs\") pod \"58e4fb66-49ec-436a-8e2a-8890f83ed081\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") "
Oct 29 23:01:01.981169 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.980921    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-model-cache" pod "62e206a5-c7f6-4865-98bd-ca3a68f5312a" (UID: "62e206a5-c7f6-4865-98bd-ca3a68f5312a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:01.981169 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.980925    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-model-cache" pod "58e4fb66-49ec-436a-8e2a-8890f83ed081" (UID: "58e4fb66-49ec-436a-8e2a-8890f83ed081"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:01.981169 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.981039    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-dshm\") pod \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") "
Oct 29 23:01:01.981169 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.981117    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-dshm\") pod \"58e4fb66-49ec-436a-8e2a-8890f83ed081\" (UID: \"58e4fb66-49ec-436a-8e2a-8890f83ed081\") "
Oct 29 23:01:01.981169 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.981153    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-tmp-dir" pod "58e4fb66-49ec-436a-8e2a-8890f83ed081" (UID: "58e4fb66-49ec-436a-8e2a-8890f83ed081"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:01.981436 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.981208    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e206a5-c7f6-4865-98bd-ca3a68f5312a-kube-api-access-tpjvd\" (UniqueName: \"kubernetes.io/projected/62e206a5-c7f6-4865-98bd-ca3a68f5312a-kube-api-access-tpjvd\") pod \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\" (UID: \"62e206a5-c7f6-4865-98bd-ca3a68f5312a\") "
Oct 29 23:01:01.981436 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.981390    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-home" pod "58e4fb66-49ec-436a-8e2a-8890f83ed081" (UID: "58e4fb66-49ec-436a-8e2a-8890f83ed081"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:01.982182 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.982160    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:01.982311 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.982186    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:01.982311 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.982195    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:01.982311 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.982203    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:01.982311 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.982212    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:01.982765 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.982743    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tmp-dir" pod "62e206a5-c7f6-4865-98bd-ca3a68f5312a" (UID: "62e206a5-c7f6-4865-98bd-ca3a68f5312a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:01.984848 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.984820    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-dshm" pod "62e206a5-c7f6-4865-98bd-ca3a68f5312a" (UID: "62e206a5-c7f6-4865-98bd-ca3a68f5312a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:01.984848 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.984833    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-dshm" pod "58e4fb66-49ec-436a-8e2a-8890f83ed081" (UID: "58e4fb66-49ec-436a-8e2a-8890f83ed081"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:01.985621 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.985589    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tls-certs" pod "62e206a5-c7f6-4865-98bd-ca3a68f5312a" (UID: "62e206a5-c7f6-4865-98bd-ca3a68f5312a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:01.994217 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.994188    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e4fb66-49ec-436a-8e2a-8890f83ed081-tls-certs" pod "58e4fb66-49ec-436a-8e2a-8890f83ed081" (UID: "58e4fb66-49ec-436a-8e2a-8890f83ed081"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:01.994474 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.994449    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e206a5-c7f6-4865-98bd-ca3a68f5312a-kube-api-access-tpjvd" pod "62e206a5-c7f6-4865-98bd-ca3a68f5312a" (UID: "62e206a5-c7f6-4865-98bd-ca3a68f5312a"). InnerVolumeSpecName "kube-api-access-tpjvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:01.997005 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:01.996978    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e4fb66-49ec-436a-8e2a-8890f83ed081-kube-api-access-2pgqt" pod "58e4fb66-49ec-436a-8e2a-8890f83ed081" (UID: "58e4fb66-49ec-436a-8e2a-8890f83ed081"). InnerVolumeSpecName "kube-api-access-2pgqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:02.084035 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.084002    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpjvd\" (UniqueName: \"kubernetes.io/projected/62e206a5-c7f6-4865-98bd-ca3a68f5312a-kube-api-access-tpjvd\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.084035 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.084035    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pgqt\" (UniqueName: \"kubernetes.io/projected/58e4fb66-49ec-436a-8e2a-8890f83ed081-kube-api-access-2pgqt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.084288 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.084049    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.084288 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.084062    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e206a5-c7f6-4865-98bd-ca3a68f5312a-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.084288 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.084075    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e4fb66-49ec-436a-8e2a-8890f83ed081-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.084288 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.084085    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e206a5-c7f6-4865-98bd-ca3a68f5312a-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.084288 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.084095    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e4fb66-49ec-436a-8e2a-8890f83ed081-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.123558 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.123477    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70fe7a3-9e71-4add-948e-815c8ef7d84a" path="/var/lib/kubelet/pods/f70fe7a3-9e71-4add-948e-815c8ef7d84a/volumes"
Oct 29 23:01:02.448679 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.448644    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:02.899072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.898993    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"
Oct 29 23:01:02.899072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.899013    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw" event={"ID":"62e206a5-c7f6-4865-98bd-ca3a68f5312a","Type":"ContainerDied","Data":"805a631c2d295701b968303c816d02e5e00424d69cd838234e60c9aa4c9cb351"}
Oct 29 23:01:02.899072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.899068    4860 scope.go:122] "RemoveContainer" containerID="19a2ffaea059c86af3276569651c889b91a589a77ab2640a4ef02875b14d917f"
Oct 29 23:01:02.901101 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.901082    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"
Oct 29 23:01:02.901223 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.901092    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6" event={"ID":"58e4fb66-49ec-436a-8e2a-8890f83ed081","Type":"ContainerDied","Data":"f42a5ecad1584cd50cd027641e66178a332dc0b6e335584ca4f82a9845173f9d"}
Oct 29 23:01:02.912052 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.912032    4860 scope.go:122] "RemoveContainer" containerID="4a8dedc61c05bdbeb3d0547296b96802cab77c456df133b67650cceef06aa582"
Oct 29 23:01:02.957963 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.957933    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"]
Oct 29 23:01:02.971518 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.971494    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/simulator-w3-43b37d-kserve-5d86fc59fc-gwqd6"]
Oct 29 23:01:02.999539 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:02.999514    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"]
Oct 29 23:01:03.007193 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.007171    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-43b37d/distinct-2-w3-43b37d-kserve-587895fc96-tmhsw"]
Oct 29 23:01:03.491334 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.491294    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 23:01:03.609317 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609289    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-credential-socket\") pod \"bbc22e15-7269-4307-a9bf-ad39641f7287\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") "
Oct 29 23:01:03.609612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609380    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-socket\") pod \"bbc22e15-7269-4307-a9bf-ad39641f7287\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") "
Oct 29 23:01:03.609612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609408    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-envoy\") pod \"bbc22e15-7269-4307-a9bf-ad39641f7287\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") "
Oct 29 23:01:03.609612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609489    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc22e15-7269-4307-a9bf-ad39641f7287-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc22e15-7269-4307-a9bf-ad39641f7287-istiod-ca-cert\") pod \"bbc22e15-7269-4307-a9bf-ad39641f7287\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") "
Oct 29 23:01:03.609612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609535    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-credential-socket" pod "bbc22e15-7269-4307-a9bf-ad39641f7287" (UID: "bbc22e15-7269-4307-a9bf-ad39641f7287"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.609612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609551    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-certs\") pod \"bbc22e15-7269-4307-a9bf-ad39641f7287\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") "
Oct 29 23:01:03.609932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609697    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-podinfo\") pod \"bbc22e15-7269-4307-a9bf-ad39641f7287\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") "
Oct 29 23:01:03.609932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609742    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-token\" (UniqueName: \"kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-token\") pod \"bbc22e15-7269-4307-a9bf-ad39641f7287\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") "
Oct 29 23:01:03.609932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609683    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-socket" pod "bbc22e15-7269-4307-a9bf-ad39641f7287" (UID: "bbc22e15-7269-4307-a9bf-ad39641f7287"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.609932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609752    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-certs" pod "bbc22e15-7269-4307-a9bf-ad39641f7287" (UID: "bbc22e15-7269-4307-a9bf-ad39641f7287"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.609932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609810    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-kube-api-access-4d45t\" (UniqueName: \"kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-kube-api-access-4d45t\") pod \"bbc22e15-7269-4307-a9bf-ad39641f7287\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") "
Oct 29 23:01:03.609932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609869    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-data\") pod \"bbc22e15-7269-4307-a9bf-ad39641f7287\" (UID: \"bbc22e15-7269-4307-a9bf-ad39641f7287\") "
Oct 29 23:01:03.610272 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.609937    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc22e15-7269-4307-a9bf-ad39641f7287-istiod-ca-cert" pod "bbc22e15-7269-4307-a9bf-ad39641f7287" (UID: "bbc22e15-7269-4307-a9bf-ad39641f7287"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:03.610603 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.610415    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-data" pod "bbc22e15-7269-4307-a9bf-ad39641f7287" (UID: "bbc22e15-7269-4307-a9bf-ad39641f7287"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.611897 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.611841    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.612003 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.611906    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.612003 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.611936    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc22e15-7269-4307-a9bf-ad39641f7287-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.612003 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.611954    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.612003 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.611967    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.612233 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.612201    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-podinfo" pod "bbc22e15-7269-4307-a9bf-ad39641f7287" (UID: "bbc22e15-7269-4307-a9bf-ad39641f7287"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:03.612596 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.612569    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-token" pod "bbc22e15-7269-4307-a9bf-ad39641f7287" (UID: "bbc22e15-7269-4307-a9bf-ad39641f7287"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:03.612690 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.612625    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-envoy" pod "bbc22e15-7269-4307-a9bf-ad39641f7287" (UID: "bbc22e15-7269-4307-a9bf-ad39641f7287"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.612917 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.612884    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-kube-api-access-4d45t" pod "bbc22e15-7269-4307-a9bf-ad39641f7287" (UID: "bbc22e15-7269-4307-a9bf-ad39641f7287"). InnerVolumeSpecName "kube-api-access-4d45t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:03.713509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.713454    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.713509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.713481    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.713509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.713495    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.713509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.713507    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d45t\" (UniqueName: \"kubernetes.io/projected/bbc22e15-7269-4307-a9bf-ad39641f7287-kube-api-access-4d45t\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.915197 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.915117    4860 generic.go:356] "Generic (PLEG): container finished" podID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerID="6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0" exitCode=0
Oct 29 23:01:03.915197 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.915178    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"
Oct 29 23:01:03.915720 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.915197    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" event={"ID":"bbc22e15-7269-4307-a9bf-ad39641f7287","Type":"ContainerDied","Data":"6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0"}
Oct 29 23:01:03.915720 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.915250    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l" event={"ID":"bbc22e15-7269-4307-a9bf-ad39641f7287","Type":"ContainerDied","Data":"4bfa3eb67207460a9119517981670c7d5b8a4198a881ca98166417c024fc904b"}
Oct 29 23:01:03.915720 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.915277    4860 scope.go:122] "RemoveContainer" containerID="6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0"
Oct 29 23:01:03.920314 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.920289    4860 generic.go:356] "Generic (PLEG): container finished" podID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerID="fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6" exitCode=0
Oct 29 23:01:03.920692 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.920496    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" event={"ID":"8d139364-f919-4189-b3fc-cf72ade3e6d6","Type":"ContainerDied","Data":"fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6"}
Oct 29 23:01:03.920925 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.920899    4860 scope.go:122] "RemoveContainer" containerID="fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6"
Oct 29 23:01:03.923471 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.923419    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:01:03.923586 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.923522    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:01:03.923586 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:03.923576    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:01:04.012349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.009771    4860 scope.go:122] "RemoveContainer" containerID="53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c"
Oct 29 23:01:04.163738 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.161712    4860 scope.go:122] "RemoveContainer" containerID="6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0"
Oct 29 23:01:04.166297 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:04.166161    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0\": container with ID starting with 6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0 not found: ID does not exist" containerID="6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0"
Oct 29 23:01:04.166297 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.166198    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0"} err="failed to get container status \"6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0\": rpc error: code = NotFound desc = could not find container \"6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0\": container with ID starting with 6cef852ebf68cfda3a859898bcbdfcd1fa0527a5ca06c0fc014ab4878b1648a0 not found: ID does not exist"
Oct 29 23:01:04.166297 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.166224    4860 scope.go:122] "RemoveContainer" containerID="53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c"
Oct 29 23:01:04.174496 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.170016    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" path="/var/lib/kubelet/pods/58e4fb66-49ec-436a-8e2a-8890f83ed081/volumes"
Oct 29 23:01:04.174496 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.170691    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" path="/var/lib/kubelet/pods/62e206a5-c7f6-4865-98bd-ca3a68f5312a/volumes"
Oct 29 23:01:04.175085 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:04.175061    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c\": container with ID starting with 53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c not found: ID does not exist" containerID="53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c"
Oct 29 23:01:04.175229 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.175206    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c"} err="failed to get container status \"53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c\": rpc error: code = NotFound desc = could not find container \"53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c\": container with ID starting with 53572f86e520c5da40a6c50a243ca6d936a7b141351052dfdef42e325feb3b5c not found: ID does not exist"
Oct 29 23:01:04.180365 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.180317    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"]
Oct 29 23:01:04.189181 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.189148    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-784e7a-openshift-default-595898bd5c-tf78l"]
Oct 29 23:01:04.461977 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.456961    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:04.664342 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.664302    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw" podUID="f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:04.664523 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.664302    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc" podUID="056308d8-3cce-4df1-a0a9-e0486218e4f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:04.931210 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:04.931171    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" event={"ID":"8d139364-f919-4189-b3fc-cf72ade3e6d6","Type":"ContainerStarted","Data":"61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067"}
Oct 29 23:01:05.231996 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:05.231913    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:05.554083 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:05.553985    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 23:01:05.554250 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:05.554192    4860 patch_prober.go:29] interesting pod/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Oct 29 23:01:05.554313 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:05.554254    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Oct 29 23:01:05.665839 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:05.665805    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:05.798103 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:05.792170    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:05.952346 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:05.948762    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:05.992350 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:05.991820    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"]
Oct 29 23:01:05.992553 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:05.992110    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx" podUID="88824988-8fe2-4a1b-af86-31647faa8af2" containerName="istio-proxy" containerID="cri-o://fc20f291fe9168359b92731ad417dfc8a7ddfded1b3bff86cbabce60bb0bcb02" gracePeriod=30
Oct 29 23:01:06.035352 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:06.035293    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"]
Oct 29 23:01:06.035803 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:06.035765    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w" podUID="f864ec3f-6650-4081-9656-cb8f9009b4f0" containerName="istio-proxy" containerID="cri-o://984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330" gracePeriod=30
Oct 29 23:01:06.118357 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:06.114357    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:06.319052 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:06.316347    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" path="/var/lib/kubelet/pods/bbc22e15-7269-4307-a9bf-ad39641f7287/volumes"
Oct 29 23:01:06.320287 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:06.320144    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:06.530384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:06.530138    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:06.613120 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:06.613039    4860 patch_prober.go:29] interesting pod/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Oct 29 23:01:06.613120 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:06.613097    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Oct 29 23:01:06.706898 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:06.706860    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 23:01:07.553598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:07.553565    4860 patch_prober.go:29] interesting pod/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Oct 29 23:01:07.554006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:07.553615    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Oct 29 23:01:08.244144 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:08.244090    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:08.553908 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:08.553829    4860 patch_prober.go:29] interesting pod/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Oct 29 23:01:08.553908 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:08.553891    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Oct 29 23:01:08.897660 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:08.897565    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:09.016371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:09.016306    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:09.511634 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:09.511579    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:09.553842 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:09.553805    4860 patch_prober.go:29] interesting pod/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused" start-of-body=
Oct 29 23:01:09.554007 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:09.553871    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.84:15021/healthz/ready\": dial tcp 10.132.1.84:15021: connect: connection refused"
Oct 29 23:01:09.680348 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:09.679975    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:10.602318 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:10.602281    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:10.898718 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:10.898606    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 23:01:10.901858 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:10.901836    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 23:01:11.071080 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.071052    4860 generic.go:356] "Generic (PLEG): container finished" podID="88824988-8fe2-4a1b-af86-31647faa8af2" containerID="fc20f291fe9168359b92731ad417dfc8a7ddfded1b3bff86cbabce60bb0bcb02" exitCode=0
Oct 29 23:01:11.071174 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.071123    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx" event={"ID":"88824988-8fe2-4a1b-af86-31647faa8af2","Type":"ContainerDied","Data":"fc20f291fe9168359b92731ad417dfc8a7ddfded1b3bff86cbabce60bb0bcb02"}
Oct 29 23:01:11.283862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.283835    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 23:01:11.349282 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.349249    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-envoy\") pod \"88824988-8fe2-4a1b-af86-31647faa8af2\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") "
Oct 29 23:01:11.349455 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.349311    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-certs\") pod \"88824988-8fe2-4a1b-af86-31647faa8af2\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") "
Oct 29 23:01:11.349455 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.349415    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88824988-8fe2-4a1b-af86-31647faa8af2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88824988-8fe2-4a1b-af86-31647faa8af2-istiod-ca-cert\") pod \"88824988-8fe2-4a1b-af86-31647faa8af2\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") "
Oct 29 23:01:11.349562 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.349469    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-istio-token\" (UniqueName: \"kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-istio-token\") pod \"88824988-8fe2-4a1b-af86-31647faa8af2\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") "
Oct 29 23:01:11.349562 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.349521    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-kube-api-access-zr9kc\" (UniqueName: \"kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-kube-api-access-zr9kc\") pod \"88824988-8fe2-4a1b-af86-31647faa8af2\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") "
Oct 29 23:01:11.349691 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.349575    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-data\") pod \"88824988-8fe2-4a1b-af86-31647faa8af2\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") "
Oct 29 23:01:11.349691 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.349676    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88824988-8fe2-4a1b-af86-31647faa8af2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88824988-8fe2-4a1b-af86-31647faa8af2-istio-podinfo\") pod \"88824988-8fe2-4a1b-af86-31647faa8af2\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") "
Oct 29 23:01:11.349801 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.349746    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-credential-socket\") pod \"88824988-8fe2-4a1b-af86-31647faa8af2\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") "
Oct 29 23:01:11.349865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.349804    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-socket\") pod \"88824988-8fe2-4a1b-af86-31647faa8af2\" (UID: \"88824988-8fe2-4a1b-af86-31647faa8af2\") "
Oct 29 23:01:11.350017 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.349673    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-certs" pod "88824988-8fe2-4a1b-af86-31647faa8af2" (UID: "88824988-8fe2-4a1b-af86-31647faa8af2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.350092 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.349914    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88824988-8fe2-4a1b-af86-31647faa8af2-istiod-ca-cert" pod "88824988-8fe2-4a1b-af86-31647faa8af2" (UID: "88824988-8fe2-4a1b-af86-31647faa8af2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.350371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.350275    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-credential-socket" pod "88824988-8fe2-4a1b-af86-31647faa8af2" (UID: "88824988-8fe2-4a1b-af86-31647faa8af2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.350371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.350312    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-socket" pod "88824988-8fe2-4a1b-af86-31647faa8af2" (UID: "88824988-8fe2-4a1b-af86-31647faa8af2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.350576 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.350558    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.350647 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.350580    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.350647 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.350590    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.350647 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.350598    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88824988-8fe2-4a1b-af86-31647faa8af2-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.351054 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.351031    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-data" pod "88824988-8fe2-4a1b-af86-31647faa8af2" (UID: "88824988-8fe2-4a1b-af86-31647faa8af2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.351911 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.351877    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-envoy" pod "88824988-8fe2-4a1b-af86-31647faa8af2" (UID: "88824988-8fe2-4a1b-af86-31647faa8af2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.352480 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.352293    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-kube-api-access-zr9kc" pod "88824988-8fe2-4a1b-af86-31647faa8af2" (UID: "88824988-8fe2-4a1b-af86-31647faa8af2"). InnerVolumeSpecName "kube-api-access-zr9kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.353108 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.353078    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-istio-token" pod "88824988-8fe2-4a1b-af86-31647faa8af2" (UID: "88824988-8fe2-4a1b-af86-31647faa8af2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.353474 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.353450    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88824988-8fe2-4a1b-af86-31647faa8af2-istio-podinfo" pod "88824988-8fe2-4a1b-af86-31647faa8af2" (UID: "88824988-8fe2-4a1b-af86-31647faa8af2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:11.371819 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.371794    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 23:01:11.451730 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.451695    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-kube-api-access-wlsgj\" (UniqueName: \"kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-kube-api-access-wlsgj\") pod \"f864ec3f-6650-4081-9656-cb8f9009b4f0\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") "
Oct 29 23:01:11.451998 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.451946    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-socket\") pod \"f864ec3f-6650-4081-9656-cb8f9009b4f0\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") "
Oct 29 23:01:11.452102 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.452084    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-credential-socket\") pod \"f864ec3f-6650-4081-9656-cb8f9009b4f0\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") "
Oct 29 23:01:11.452367 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.452280    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-socket" pod "f864ec3f-6650-4081-9656-cb8f9009b4f0" (UID: "f864ec3f-6650-4081-9656-cb8f9009b4f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.452496 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.452367    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-credential-socket" pod "f864ec3f-6650-4081-9656-cb8f9009b4f0" (UID: "f864ec3f-6650-4081-9656-cb8f9009b4f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.452602 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.452508    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-token\" (UniqueName: \"kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-token\") pod \"f864ec3f-6650-4081-9656-cb8f9009b4f0\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") "
Oct 29 23:01:11.452850 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.452830    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-podinfo\") pod \"f864ec3f-6650-4081-9656-cb8f9009b4f0\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") "
Oct 29 23:01:11.453149 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.453129    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-certs\") pod \"f864ec3f-6650-4081-9656-cb8f9009b4f0\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") "
Oct 29 23:01:11.453300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.453284    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f864ec3f-6650-4081-9656-cb8f9009b4f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f864ec3f-6650-4081-9656-cb8f9009b4f0-istiod-ca-cert\") pod \"f864ec3f-6650-4081-9656-cb8f9009b4f0\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") "
Oct 29 23:01:11.453482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.453464    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-envoy\") pod \"f864ec3f-6650-4081-9656-cb8f9009b4f0\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") "
Oct 29 23:01:11.453731 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.453710    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-data\") pod \"f864ec3f-6650-4081-9656-cb8f9009b4f0\" (UID: \"f864ec3f-6650-4081-9656-cb8f9009b4f0\") "
Oct 29 23:01:11.454929 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.454036    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f864ec3f-6650-4081-9656-cb8f9009b4f0-istiod-ca-cert" pod "f864ec3f-6650-4081-9656-cb8f9009b4f0" (UID: "f864ec3f-6650-4081-9656-cb8f9009b4f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.454929 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.454134    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-certs" pod "f864ec3f-6650-4081-9656-cb8f9009b4f0" (UID: "f864ec3f-6650-4081-9656-cb8f9009b4f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.454929 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.454541    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88824988-8fe2-4a1b-af86-31647faa8af2-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.454929 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.454741    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-data" pod "f864ec3f-6650-4081-9656-cb8f9009b4f0" (UID: "f864ec3f-6650-4081-9656-cb8f9009b4f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.454929 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.454766    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-kube-api-access-wlsgj" pod "f864ec3f-6650-4081-9656-cb8f9009b4f0" (UID: "f864ec3f-6650-4081-9656-cb8f9009b4f0"). InnerVolumeSpecName "kube-api-access-wlsgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.455865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.454564    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.455865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.455100    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f864ec3f-6650-4081-9656-cb8f9009b4f0-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.455865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.455118    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.455865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.455131    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.455865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.455144    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr9kc\" (UniqueName: \"kubernetes.io/projected/88824988-8fe2-4a1b-af86-31647faa8af2-kube-api-access-zr9kc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.455865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.455158    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.455865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.455174    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88824988-8fe2-4a1b-af86-31647faa8af2-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.455865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.455186    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.456893 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.456866    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-podinfo" pod "f864ec3f-6650-4081-9656-cb8f9009b4f0" (UID: "f864ec3f-6650-4081-9656-cb8f9009b4f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:11.457169 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.457144    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-token" pod "f864ec3f-6650-4081-9656-cb8f9009b4f0" (UID: "f864ec3f-6650-4081-9656-cb8f9009b4f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.458013 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.457988    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-envoy" pod "f864ec3f-6650-4081-9656-cb8f9009b4f0" (UID: "f864ec3f-6650-4081-9656-cb8f9009b4f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.468725 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.468700    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"]
Oct 29 23:01:11.484242 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.484216    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"]
Oct 29 23:01:11.556840 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.556806    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.556840 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.556836    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.556840 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.556845    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.557147 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.556853    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f864ec3f-6650-4081-9656-cb8f9009b4f0-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.557147 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.556862    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlsgj\" (UniqueName: \"kubernetes.io/projected/f864ec3f-6650-4081-9656-cb8f9009b4f0-kube-api-access-wlsgj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.726932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:11.726848    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:12.033529 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.033451    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"]
Oct 29 23:01:12.038420 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.038393    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f264f31b-sh65t"]
Oct 29 23:01:12.082521 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.082494    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"
Oct 29 23:01:12.082687 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.082526    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx" event={"ID":"88824988-8fe2-4a1b-af86-31647faa8af2","Type":"ContainerDied","Data":"4306c15901429ba781a190bebec921a87a5c0b4556298054dd10fd64d49a5ba5"}
Oct 29 23:01:12.082687 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.082586    4860 scope.go:122] "RemoveContainer" containerID="fc20f291fe9168359b92731ad417dfc8a7ddfded1b3bff86cbabce60bb0bcb02"
Oct 29 23:01:12.084839 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.084804    4860 generic.go:356] "Generic (PLEG): container finished" podID="f864ec3f-6650-4081-9656-cb8f9009b4f0" containerID="984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330" exitCode=0
Oct 29 23:01:12.084965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.084880    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"
Oct 29 23:01:12.085065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.085037    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w" event={"ID":"f864ec3f-6650-4081-9656-cb8f9009b4f0","Type":"ContainerDied","Data":"984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330"}
Oct 29 23:01:12.085169 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.085138    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w" event={"ID":"f864ec3f-6650-4081-9656-cb8f9009b4f0","Type":"ContainerDied","Data":"86cd9fae07cd949ba87661f456dd58cbeaaf1c6164cd59e26075ccea78c39959"}
Oct 29 23:01:12.098285 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.098270    4860 scope.go:122] "RemoveContainer" containerID="984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330"
Oct 29 23:01:12.112126 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.112096    4860 scope.go:122] "RemoveContainer" containerID="984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330"
Oct 29 23:01:12.113254 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:12.113225    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330\": container with ID starting with 984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330 not found: ID does not exist" containerID="984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330"
Oct 29 23:01:12.113371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.113263    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330"} err="failed to get container status \"984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330\": rpc error: code = NotFound desc = could not find container \"984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330\": container with ID starting with 984310ad67ca41577962d16f0ab04b579b67a375fcc9798cc3ede9127c360330 not found: ID does not exist"
Oct 29 23:01:12.117387 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.117318    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213603b4-c29d-4889-925a-d13340352ba0" path="/var/lib/kubelet/pods/213603b4-c29d-4889-925a-d13340352ba0/volumes"
Oct 29 23:01:12.143579 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.143553    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"]
Oct 29 23:01:12.159340 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.159297    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-cs6kx"]
Oct 29 23:01:12.192764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.192730    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"]
Oct 29 23:01:12.198314 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.198289    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c933cb-openshift-default-9986db7d7-jtp6w"]
Oct 29 23:01:12.967624 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:12.967588    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:14.117557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:14.117517    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88824988-8fe2-4a1b-af86-31647faa8af2" path="/var/lib/kubelet/pods/88824988-8fe2-4a1b-af86-31647faa8af2/volumes"
Oct 29 23:01:14.118216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:14.118197    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f864ec3f-6650-4081-9656-cb8f9009b4f0" path="/var/lib/kubelet/pods/f864ec3f-6650-4081-9656-cb8f9009b4f0/volumes"
Oct 29 23:01:14.608359 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:14.608319    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:14.663461 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:14.663419    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc" podUID="056308d8-3cce-4df1-a0a9-e0486218e4f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:14.663682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:14.663590    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw" podUID="f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:15.098425 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:15.098394    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:15.604069 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:15.604010    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:16.459247 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:16.459216    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:17.178802 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.178775    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"]
Oct 29 23:01:17.339248 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.339211    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:17.465207 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.465147    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:17.504773 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.504747    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"
Oct 29 23:01:17.531166 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.531146    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"
Oct 29 23:01:17.650729 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.650651    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-kube-api-access-bghsc\" (UniqueName: \"kubernetes.io/projected/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-kube-api-access-bghsc\") pod \"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43\" (UID: \"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43\") "
Oct 29 23:01:17.650896 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.650741    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056308d8-3cce-4df1-a0a9-e0486218e4f5-kube-api-access-4md8k\" (UniqueName: \"kubernetes.io/projected/056308d8-3cce-4df1-a0a9-e0486218e4f5-kube-api-access-4md8k\") pod \"056308d8-3cce-4df1-a0a9-e0486218e4f5\" (UID: \"056308d8-3cce-4df1-a0a9-e0486218e4f5\") "
Oct 29 23:01:17.650979 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.650940    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-plugins-config-volume\") pod \"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43\" (UID: \"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43\") "
Oct 29 23:01:17.651038 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.650978    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056308d8-3cce-4df1-a0a9-e0486218e4f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056308d8-3cce-4df1-a0a9-e0486218e4f5-plugins-config-volume\") pod \"056308d8-3cce-4df1-a0a9-e0486218e4f5\" (UID: \"056308d8-3cce-4df1-a0a9-e0486218e4f5\") "
Oct 29 23:01:17.651406 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.651376    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-plugins-config-volume" pod "f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43" (UID: "f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:17.651509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.651420    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056308d8-3cce-4df1-a0a9-e0486218e4f5-plugins-config-volume" pod "056308d8-3cce-4df1-a0a9-e0486218e4f5" (UID: "056308d8-3cce-4df1-a0a9-e0486218e4f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:17.651674 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.651657    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:17.651716 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.651677    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056308d8-3cce-4df1-a0a9-e0486218e4f5-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:17.652868 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.652842    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-kube-api-access-bghsc" pod "f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43" (UID: "f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43"). InnerVolumeSpecName "kube-api-access-bghsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:17.652983 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.652877    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056308d8-3cce-4df1-a0a9-e0486218e4f5-kube-api-access-4md8k" pod "056308d8-3cce-4df1-a0a9-e0486218e4f5" (UID: "056308d8-3cce-4df1-a0a9-e0486218e4f5"). InnerVolumeSpecName "kube-api-access-4md8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:17.752516 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.752490    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bghsc\" (UniqueName: \"kubernetes.io/projected/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43-kube-api-access-bghsc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:17.752516 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:17.752514    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4md8k\" (UniqueName: \"kubernetes.io/projected/056308d8-3cce-4df1-a0a9-e0486218e4f5-kube-api-access-4md8k\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:18.060512 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:18.060480    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:01:18.147475 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.147443    4860 generic.go:356] "Generic (PLEG): container finished" podID="f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43" containerID="f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0" exitCode=0
Oct 29 23:01:18.147700 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.147486    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"
Oct 29 23:01:18.147700 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.147523    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw" event={"ID":"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43","Type":"ContainerDied","Data":"f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0"}
Oct 29 23:01:18.147700 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.147576    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw" event={"ID":"f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43","Type":"ContainerDied","Data":"7f373f49aca1f17aca1ea88600e33cab5c58e9f84587b2477b4bb0d8e763944a"}
Oct 29 23:01:18.147700 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.147603    4860 scope.go:122] "RemoveContainer" containerID="f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0"
Oct 29 23:01:18.149168 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.149144    4860 generic.go:356] "Generic (PLEG): container finished" podID="056308d8-3cce-4df1-a0a9-e0486218e4f5" containerID="dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa" exitCode=0
Oct 29 23:01:18.149286 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.149179    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"
Oct 29 23:01:18.149286 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.149250    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc" event={"ID":"056308d8-3cce-4df1-a0a9-e0486218e4f5","Type":"ContainerDied","Data":"dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa"}
Oct 29 23:01:18.149441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.149305    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc" event={"ID":"056308d8-3cce-4df1-a0a9-e0486218e4f5","Type":"ContainerDied","Data":"b2bbda9a0050daa85c45c9a1cc698121383d621f62d1c6150455540e029de695"}
Oct 29 23:01:18.162341 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.159933    4860 scope.go:122] "RemoveContainer" containerID="f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0"
Oct 29 23:01:18.162341 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:18.160528    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0\": container with ID starting with f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0 not found: ID does not exist" containerID="f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0"
Oct 29 23:01:18.162341 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.160556    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0"} err="failed to get container status \"f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0\": rpc error: code = NotFound desc = could not find container \"f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0\": container with ID starting with f9a3ecaea2d3cbc266081407eecf765013f8878114d5719e4d25830a664906a0 not found: ID does not exist"
Oct 29 23:01:18.162341 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.160576    4860 scope.go:122] "RemoveContainer" containerID="dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa"
Oct 29 23:01:18.172949 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.172931    4860 scope.go:122] "RemoveContainer" containerID="dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa"
Oct 29 23:01:18.173242 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:18.173221    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa\": container with ID starting with dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa not found: ID does not exist" containerID="dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa"
Oct 29 23:01:18.173331 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.173254    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa"} err="failed to get container status \"dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa\": rpc error: code = NotFound desc = could not find container \"dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa\": container with ID starting with dce315f1f6c3e0fb977da2a9459101f44b8373086087d93fc46fc5020c363ffa not found: ID does not exist"
Oct 29 23:01:18.208065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.208041    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"]
Oct 29 23:01:18.212832 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.212813    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-43b37d-fd685c7db-qkngc"]
Oct 29 23:01:18.238087 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.238062    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:18.246721 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.246699    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"]
Oct 29 23:01:18.253125 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:18.253101    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-43b37d-74d9bdb487-qtfzw"]
Oct 29 23:01:19.863118 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:19.863083    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:20.115554 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:20.115475    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056308d8-3cce-4df1-a0a9-e0486218e4f5" path="/var/lib/kubelet/pods/056308d8-3cce-4df1-a0a9-e0486218e4f5/volumes"
Oct 29 23:01:20.115946 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:20.115931    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43" path="/var/lib/kubelet/pods/f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43/volumes"
Oct 29 23:01:20.132215 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:20.132188    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"]
Oct 29 23:01:20.132425 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:20.132401    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5" podUID="27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" containerName="istio-proxy" containerID="cri-o://ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93" gracePeriod=30
Oct 29 23:01:21.751548 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:21.751517    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:22.179522 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:22.179425    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q" podUID="100f773a-93c3-49ef-b704-393f72d7b181" containerName="payload-processing" containerID="cri-o://8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931" gracePeriod=25
Oct 29 23:01:23.093285 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.093253    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"]
Oct 29 23:01:23.093824 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.093502    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c" podUID="be999b8c-3f77-451a-9d23-be1db342ce44" containerName="maas-api" containerID="cri-o://a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1" gracePeriod=30
Oct 29 23:01:23.185393 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.185303    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:23.259805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.259773    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"]
Oct 29 23:01:23.259998 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.259971    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9" podUID="130135b0-3ddc-47eb-a913-5648e0e93b8d" containerName="payload-processing" containerID="cri-o://bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf" gracePeriod=30
Oct 29 23:01:23.283289 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.283238    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"]
Oct 29 23:01:23.283727 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.283505    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5" podUID="9f4a5ad2-3f37-4854-975d-355793e94b71" containerName="payload-pre-processing" containerID="cri-o://c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe" gracePeriod=30
Oct 29 23:01:23.361592 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.361556    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:23.468301 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.468255    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:23.487821 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.487296    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 23:01:23.576247 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.576216    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"
Oct 29 23:01:23.599253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.599223    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:23.639927 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.639904    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"
Oct 29 23:01:23.641121 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.641098    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be999b8c-3f77-451a-9d23-be1db342ce44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be999b8c-3f77-451a-9d23-be1db342ce44-openshift-service-ca\") pod \"be999b8c-3f77-451a-9d23-be1db342ce44\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") "
Oct 29 23:01:23.641258 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.641238    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-metrics-certs\") pod \"be999b8c-3f77-451a-9d23-be1db342ce44\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") "
Oct 29 23:01:23.641390 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.641371    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-maas-api-tls\") pod \"be999b8c-3f77-451a-9d23-be1db342ce44\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") "
Oct 29 23:01:23.641488 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.641465    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be999b8c-3f77-451a-9d23-be1db342ce44-openshift-service-ca" pod "be999b8c-3f77-451a-9d23-be1db342ce44" (UID: "be999b8c-3f77-451a-9d23-be1db342ce44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:23.641552 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.641478    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be999b8c-3f77-451a-9d23-be1db342ce44-kube-api-access-vc7jz\" (UniqueName: \"kubernetes.io/projected/be999b8c-3f77-451a-9d23-be1db342ce44-kube-api-access-vc7jz\") pod \"be999b8c-3f77-451a-9d23-be1db342ce44\" (UID: \"be999b8c-3f77-451a-9d23-be1db342ce44\") "
Oct 29 23:01:23.642029 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.642010    4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be999b8c-3f77-451a-9d23-be1db342ce44-openshift-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:23.643493 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.643473    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-metrics-certs" pod "be999b8c-3f77-451a-9d23-be1db342ce44" (UID: "be999b8c-3f77-451a-9d23-be1db342ce44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:23.643564 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.643535    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be999b8c-3f77-451a-9d23-be1db342ce44-kube-api-access-vc7jz" pod "be999b8c-3f77-451a-9d23-be1db342ce44" (UID: "be999b8c-3f77-451a-9d23-be1db342ce44"). InnerVolumeSpecName "kube-api-access-vc7jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:23.643605 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.643562    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-maas-api-tls" pod "be999b8c-3f77-451a-9d23-be1db342ce44" (UID: "be999b8c-3f77-451a-9d23-be1db342ce44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:23.742678 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.742645    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130135b0-3ddc-47eb-a913-5648e0e93b8d-kube-api-access-9sr8k\" (UniqueName: \"kubernetes.io/projected/130135b0-3ddc-47eb-a913-5648e0e93b8d-kube-api-access-9sr8k\") pod \"130135b0-3ddc-47eb-a913-5648e0e93b8d\" (UID: \"130135b0-3ddc-47eb-a913-5648e0e93b8d\") "
Oct 29 23:01:23.742881 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.742709    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/130135b0-3ddc-47eb-a913-5648e0e93b8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130135b0-3ddc-47eb-a913-5648e0e93b8d-plugins-config-volume\") pod \"130135b0-3ddc-47eb-a913-5648e0e93b8d\" (UID: \"130135b0-3ddc-47eb-a913-5648e0e93b8d\") "
Oct 29 23:01:23.742881 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.742740    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4a5ad2-3f37-4854-975d-355793e94b71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f4a5ad2-3f37-4854-975d-355793e94b71-plugins-config-volume\") pod \"9f4a5ad2-3f37-4854-975d-355793e94b71\" (UID: \"9f4a5ad2-3f37-4854-975d-355793e94b71\") "
Oct 29 23:01:23.743032 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.742904    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4a5ad2-3f37-4854-975d-355793e94b71-kube-api-access-sk962\" (UniqueName: \"kubernetes.io/projected/9f4a5ad2-3f37-4854-975d-355793e94b71-kube-api-access-sk962\") pod \"9f4a5ad2-3f37-4854-975d-355793e94b71\" (UID: \"9f4a5ad2-3f37-4854-975d-355793e94b71\") "
Oct 29 23:01:23.743244 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.743211    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4a5ad2-3f37-4854-975d-355793e94b71-plugins-config-volume" pod "9f4a5ad2-3f37-4854-975d-355793e94b71" (UID: "9f4a5ad2-3f37-4854-975d-355793e94b71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:23.743382 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.743300    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130135b0-3ddc-47eb-a913-5648e0e93b8d-plugins-config-volume" pod "130135b0-3ddc-47eb-a913-5648e0e93b8d" (UID: "130135b0-3ddc-47eb-a913-5648e0e93b8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:23.743801 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.743774    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-maas-api-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:23.743906 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.743806    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc7jz\" (UniqueName: \"kubernetes.io/projected/be999b8c-3f77-451a-9d23-be1db342ce44-kube-api-access-vc7jz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:23.743906 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.743822    4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be999b8c-3f77-451a-9d23-be1db342ce44-metrics-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:23.743906 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.743837    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130135b0-3ddc-47eb-a913-5648e0e93b8d-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:23.743906 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.743851    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f4a5ad2-3f37-4854-975d-355793e94b71-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:23.745145 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.745119    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130135b0-3ddc-47eb-a913-5648e0e93b8d-kube-api-access-9sr8k" pod "130135b0-3ddc-47eb-a913-5648e0e93b8d" (UID: "130135b0-3ddc-47eb-a913-5648e0e93b8d"). InnerVolumeSpecName "kube-api-access-9sr8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:23.746370 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.746110    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4a5ad2-3f37-4854-975d-355793e94b71-kube-api-access-sk962" pod "9f4a5ad2-3f37-4854-975d-355793e94b71" (UID: "9f4a5ad2-3f37-4854-975d-355793e94b71"). InnerVolumeSpecName "kube-api-access-sk962". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:23.846420 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.846381    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sr8k\" (UniqueName: \"kubernetes.io/projected/130135b0-3ddc-47eb-a913-5648e0e93b8d-kube-api-access-9sr8k\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:23.846420 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.846421    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk962\" (UniqueName: \"kubernetes.io/projected/9f4a5ad2-3f37-4854-975d-355793e94b71-kube-api-access-sk962\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:23.907970 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:23.907891    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:24.027702 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.027672    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:24.168521 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.168482    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:24.207917 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.207887    4860 generic.go:356] "Generic (PLEG): container finished" podID="9f4a5ad2-3f37-4854-975d-355793e94b71" containerID="c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe" exitCode=0
Oct 29 23:01:24.208049 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.207945    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"
Oct 29 23:01:24.208049 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.207974    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5" event={"ID":"9f4a5ad2-3f37-4854-975d-355793e94b71","Type":"ContainerDied","Data":"c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe"}
Oct 29 23:01:24.208049 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.208027    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5" event={"ID":"9f4a5ad2-3f37-4854-975d-355793e94b71","Type":"ContainerDied","Data":"3b768c4fde3060505119fbaf92e9da7b3256ba71eb1a2ec9496c47ea1527eb0a"}
Oct 29 23:01:24.208049 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.208043    4860 scope.go:122] "RemoveContainer" containerID="c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe"
Oct 29 23:01:24.210023 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.209996    4860 generic.go:356] "Generic (PLEG): container finished" podID="130135b0-3ddc-47eb-a913-5648e0e93b8d" containerID="bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf" exitCode=0
Oct 29 23:01:24.210129 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.210089    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9" event={"ID":"130135b0-3ddc-47eb-a913-5648e0e93b8d","Type":"ContainerDied","Data":"bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf"}
Oct 29 23:01:24.210184 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.210129    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9" event={"ID":"130135b0-3ddc-47eb-a913-5648e0e93b8d","Type":"ContainerDied","Data":"dd6ce954dbcba5383f902993ac277c333b3b2b24c4ecf65dcb9550a7a003088c"}
Oct 29 23:01:24.210230 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.210218    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"
Oct 29 23:01:24.212531 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.212497    4860 generic.go:356] "Generic (PLEG): container finished" podID="be999b8c-3f77-451a-9d23-be1db342ce44" containerID="a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1" exitCode=0
Oct 29 23:01:24.212650 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.212557    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"
Oct 29 23:01:24.212797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.212773    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c" event={"ID":"be999b8c-3f77-451a-9d23-be1db342ce44","Type":"ContainerDied","Data":"a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1"}
Oct 29 23:01:24.212934 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.212821    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c" event={"ID":"be999b8c-3f77-451a-9d23-be1db342ce44","Type":"ContainerDied","Data":"ff83f37f725db575e739ea67bcfbd1514232fff0d7c6ce787c528aa7e5851ec9"}
Oct 29 23:01:24.220940 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.220924    4860 scope.go:122] "RemoveContainer" containerID="c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe"
Oct 29 23:01:24.221218 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:24.221198    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe\": container with ID starting with c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe not found: ID does not exist" containerID="c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe"
Oct 29 23:01:24.221286 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.221226    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe"} err="failed to get container status \"c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe\": rpc error: code = NotFound desc = could not find container \"c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe\": container with ID starting with c4e4e6df11815b1ec524ac82777b33125ded86d28a7a39d2e449c45ab6f2affe not found: ID does not exist"
Oct 29 23:01:24.221286 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.221244    4860 scope.go:122] "RemoveContainer" containerID="bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf"
Oct 29 23:01:24.232154 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.232133    4860 scope.go:122] "RemoveContainer" containerID="bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf"
Oct 29 23:01:24.232534 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:24.232497    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf\": container with ID starting with bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf not found: ID does not exist" containerID="bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf"
Oct 29 23:01:24.232695 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.232666    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf"} err="failed to get container status \"bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf\": rpc error: code = NotFound desc = could not find container \"bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf\": container with ID starting with bef80e243962ec893185d1efe2dc55523852e6a229013bd258a742e04088bccf not found: ID does not exist"
Oct 29 23:01:24.232798 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.232696    4860 scope.go:122] "RemoveContainer" containerID="a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1"
Oct 29 23:01:24.246031 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.245888    4860 scope.go:122] "RemoveContainer" containerID="a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1"
Oct 29 23:01:24.246179 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:24.246157    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1\": container with ID starting with a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1 not found: ID does not exist" containerID="a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1"
Oct 29 23:01:24.246253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.246189    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1"} err="failed to get container status \"a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1\": rpc error: code = NotFound desc = could not find container \"a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1\": container with ID starting with a3c05e72cb13b5a11fa5f0f5b6e30f3bd3a87394b587104cca42e57018019fd1 not found: ID does not exist"
Oct 29 23:01:24.262999 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.262977    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"]
Oct 29 23:01:24.263195 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.263174    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb" podUID="f91089d4-1a38-4d94-87f8-8131ab078130" containerName="payload-processing" containerID="cri-o://eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c" gracePeriod=30
Oct 29 23:01:24.268609 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.268587    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"]
Oct 29 23:01:24.277634 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.277610    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-83f120-f467f9994-b9pr9"]
Oct 29 23:01:24.288060 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.288039    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:24.323794 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.323744    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"]
Oct 29 23:01:24.328722 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.328696    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-83f120-74f9879cdc-p4v8c"]
Oct 29 23:01:24.366685 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.366648    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"]
Oct 29 23:01:24.371794 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.371764    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-83f120-75b867c88c-dcmh5"]
Oct 29 23:01:24.410865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.410840    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:24.474219 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.474190    4860 scope.go:122] "RemoveContainer" containerID="b6fe6971cdd40d36fb072575c04363302f4729a4eda70c0df00c18aa4fec1f8a"
Oct 29 23:01:24.535053 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.535027    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:24.566874 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.566851    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"
Oct 29 23:01:24.707408 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.701276    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91089d4-1a38-4d94-87f8-8131ab078130-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91089d4-1a38-4d94-87f8-8131ab078130-plugins-config-volume\") pod \"f91089d4-1a38-4d94-87f8-8131ab078130\" (UID: \"f91089d4-1a38-4d94-87f8-8131ab078130\") "
Oct 29 23:01:24.707408 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.701486    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91089d4-1a38-4d94-87f8-8131ab078130-kube-api-access-prvqg\" (UniqueName: \"kubernetes.io/projected/f91089d4-1a38-4d94-87f8-8131ab078130-kube-api-access-prvqg\") pod \"f91089d4-1a38-4d94-87f8-8131ab078130\" (UID: \"f91089d4-1a38-4d94-87f8-8131ab078130\") "
Oct 29 23:01:24.714032 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.713224    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91089d4-1a38-4d94-87f8-8131ab078130-plugins-config-volume" pod "f91089d4-1a38-4d94-87f8-8131ab078130" (UID: "f91089d4-1a38-4d94-87f8-8131ab078130"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:24.736383 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.731595    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91089d4-1a38-4d94-87f8-8131ab078130-kube-api-access-prvqg" pod "f91089d4-1a38-4d94-87f8-8131ab078130" (UID: "f91089d4-1a38-4d94-87f8-8131ab078130"). InnerVolumeSpecName "kube-api-access-prvqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:24.803414 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.803383    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91089d4-1a38-4d94-87f8-8131ab078130-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.803414 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.803411    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prvqg\" (UniqueName: \"kubernetes.io/projected/f91089d4-1a38-4d94-87f8-8131ab078130-kube-api-access-prvqg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:24.918355 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:24.918311    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:25.140997 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.140921    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:25.225588 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.225549    4860 generic.go:356] "Generic (PLEG): container finished" podID="f91089d4-1a38-4d94-87f8-8131ab078130" containerID="eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c" exitCode=0
Oct 29 23:01:25.225989 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.225598    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"
Oct 29 23:01:25.225989 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.225634    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb" event={"ID":"f91089d4-1a38-4d94-87f8-8131ab078130","Type":"ContainerDied","Data":"eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c"}
Oct 29 23:01:25.225989 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.225687    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb" event={"ID":"f91089d4-1a38-4d94-87f8-8131ab078130","Type":"ContainerDied","Data":"a2d034434e92a46b2a8997f975ca355056fa74f4eaac5142e1047ec23a1315fe"}
Oct 29 23:01:25.225989 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.225725    4860 scope.go:122] "RemoveContainer" containerID="eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c"
Oct 29 23:01:25.248822 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.248610    4860 scope.go:122] "RemoveContainer" containerID="eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c"
Oct 29 23:01:25.249490 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:25.249467    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c\": container with ID starting with eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c not found: ID does not exist" containerID="eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c"
Oct 29 23:01:25.249596 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.249500    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c"} err="failed to get container status \"eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c\": rpc error: code = NotFound desc = could not find container \"eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c\": container with ID starting with eb518fa23d0edf1908196d5552b1b688123d3a5bcc0988f66c758a286269681c not found: ID does not exist"
Oct 29 23:01:25.296943 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.296905    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:25.301387 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.301359    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"]
Oct 29 23:01:25.307783 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.307385    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7be71d-85b85b8ffd-d77xb"]
Oct 29 23:01:25.426400 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.424971    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:25.549119 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.545375    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:25.665537 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.665486    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:25.674391 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.674369    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 23:01:25.690223 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690199    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istiod-ca-cert\") pod \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") "
Oct 29 23:01:25.690369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690241    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-podinfo\") pod \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") "
Oct 29 23:01:25.690369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690282    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-socket\") pod \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") "
Oct 29 23:01:25.690511 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690451    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-credential-socket\") pod \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") "
Oct 29 23:01:25.690601 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690581    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istiod-ca-cert" pod "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" (UID: "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:25.690671 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690602    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-data\") pod \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") "
Oct 29 23:01:25.690671 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690643    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-token\" (UniqueName: \"kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-token\") pod \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") "
Oct 29 23:01:25.690671 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690644    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-socket" pod "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" (UID: "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:25.690846 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690689    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-certs\") pod \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") "
Oct 29 23:01:25.690846 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690694    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-credential-socket" pod "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" (UID: "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:25.690846 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690722    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-kube-api-access-54sb9\" (UniqueName: \"kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-kube-api-access-54sb9\") pod \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") "
Oct 29 23:01:25.690846 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690770    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-envoy\") pod \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\" (UID: \"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc\") "
Oct 29 23:01:25.690846 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690819    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-data" pod "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" (UID: "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:25.691244 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.690883    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-certs" pod "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" (UID: "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:25.691404 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.691378    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.691531 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.691406    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.691531 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.691421    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.691531 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.691434    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.691531 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.691448    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.693084 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.693060    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-envoy" pod "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" (UID: "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:25.693250 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.693228    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-podinfo" pod "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" (UID: "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:25.693294 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.693249    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-token" pod "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" (UID: "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:25.693294 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.693275    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-kube-api-access-54sb9" pod "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" (UID: "27fff8e8-73e3-4a52-8f3b-032ea9abdfcc"). InnerVolumeSpecName "kube-api-access-54sb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:25.786693 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.786647    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:25.793220 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.793195    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.793220 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.793220    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.793397 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.793229    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54sb9\" (UniqueName: \"kubernetes.io/projected/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-kube-api-access-54sb9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:25.793397 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:25.793248    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:26.115920 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.115840    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="130135b0-3ddc-47eb-a913-5648e0e93b8d" path="/var/lib/kubelet/pods/130135b0-3ddc-47eb-a913-5648e0e93b8d/volumes"
Oct 29 23:01:26.116465 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.116448    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4a5ad2-3f37-4854-975d-355793e94b71" path="/var/lib/kubelet/pods/9f4a5ad2-3f37-4854-975d-355793e94b71/volumes"
Oct 29 23:01:26.117032 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.117015    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be999b8c-3f77-451a-9d23-be1db342ce44" path="/var/lib/kubelet/pods/be999b8c-3f77-451a-9d23-be1db342ce44/volumes"
Oct 29 23:01:26.117948 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.117930    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91089d4-1a38-4d94-87f8-8131ab078130" path="/var/lib/kubelet/pods/f91089d4-1a38-4d94-87f8-8131ab078130/volumes"
Oct 29 23:01:26.235797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.235759    4860 generic.go:356] "Generic (PLEG): container finished" podID="27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" containerID="ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93" exitCode=0
Oct 29 23:01:26.236217 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.235816    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"
Oct 29 23:01:26.236217 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.235840    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5" event={"ID":"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc","Type":"ContainerDied","Data":"ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93"}
Oct 29 23:01:26.236217 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.235896    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5" event={"ID":"27fff8e8-73e3-4a52-8f3b-032ea9abdfcc","Type":"ContainerDied","Data":"4a743a5b408685fded77319dbafbdf898ae555b108aed0e41d1119de60f05f99"}
Oct 29 23:01:26.236217 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.235917    4860 scope.go:122] "RemoveContainer" containerID="ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93"
Oct 29 23:01:26.247196 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.247172    4860 scope.go:122] "RemoveContainer" containerID="ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93"
Oct 29 23:01:26.247502 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:26.247479    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93\": container with ID starting with ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93 not found: ID does not exist" containerID="ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93"
Oct 29 23:01:26.247601 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.247509    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93"} err="failed to get container status \"ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93\": rpc error: code = NotFound desc = could not find container \"ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93\": container with ID starting with ca59dbd3e41a2b5a5010b91b10ba8ece9a1259f1fecedb128c730f8560a9ef93 not found: ID does not exist"
Oct 29 23:01:26.295425 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.295403    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"]
Oct 29 23:01:26.310772 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.310744    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-eb02df-openshift-default-5c45c545c-mbzm5"]
Oct 29 23:01:26.496174 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.496136    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main" containerID="cri-o://df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e" gracePeriod=45
Oct 29 23:01:26.505027 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.504996    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main" containerID="cri-o://fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f" gracePeriod=45
Oct 29 23:01:26.789109 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.789075    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:26.791312 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.791288    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 23:01:26.806003 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.805977    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c03c61aa-d816-451a-80f0-c636cf8800a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c03c61aa-d816-451a-80f0-c636cf8800a6-tls-certs\") pod \"c03c61aa-d816-451a-80f0-c636cf8800a6\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") "
Oct 29 23:01:26.806097 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.806032    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-dshm\") pod \"c03c61aa-d816-451a-80f0-c636cf8800a6\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") "
Oct 29 23:01:26.806149 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.806123    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-model-cache\") pod \"c03c61aa-d816-451a-80f0-c636cf8800a6\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") "
Oct 29 23:01:26.806218 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.806200    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-home\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-home\") pod \"c03c61aa-d816-451a-80f0-c636cf8800a6\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") "
Oct 29 23:01:26.806401 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.806377    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03c61aa-d816-451a-80f0-c636cf8800a6-kube-api-access-b6gqq\" (UniqueName: \"kubernetes.io/projected/c03c61aa-d816-451a-80f0-c636cf8800a6-kube-api-access-b6gqq\") pod \"c03c61aa-d816-451a-80f0-c636cf8800a6\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") "
Oct 29 23:01:26.806537 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.806433    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-tmp-dir\") pod \"c03c61aa-d816-451a-80f0-c636cf8800a6\" (UID: \"c03c61aa-d816-451a-80f0-c636cf8800a6\") "
Oct 29 23:01:26.806945 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.806723    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-home" pod "c03c61aa-d816-451a-80f0-c636cf8800a6" (UID: "c03c61aa-d816-451a-80f0-c636cf8800a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:26.807151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.807092    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-tmp-dir" pod "c03c61aa-d816-451a-80f0-c636cf8800a6" (UID: "c03c61aa-d816-451a-80f0-c636cf8800a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:26.807151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.807101    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:26.807300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.807170    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-model-cache" pod "c03c61aa-d816-451a-80f0-c636cf8800a6" (UID: "c03c61aa-d816-451a-80f0-c636cf8800a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:26.812813 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.812786    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-dshm" pod "c03c61aa-d816-451a-80f0-c636cf8800a6" (UID: "c03c61aa-d816-451a-80f0-c636cf8800a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:26.812915 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.812841    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03c61aa-d816-451a-80f0-c636cf8800a6-kube-api-access-b6gqq" pod "c03c61aa-d816-451a-80f0-c636cf8800a6" (UID: "c03c61aa-d816-451a-80f0-c636cf8800a6"). InnerVolumeSpecName "kube-api-access-b6gqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:26.813864 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.813843    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c03c61aa-d816-451a-80f0-c636cf8800a6-tls-certs" pod "c03c61aa-d816-451a-80f0-c636cf8800a6" (UID: "c03c61aa-d816-451a-80f0-c636cf8800a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:26.816954 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.816936    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 23:01:26.907669 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.907640    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e631530-8524-445b-90da-4ff35c1c7f54-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e631530-8524-445b-90da-4ff35c1c7f54-tls-certs\") pod \"3e631530-8524-445b-90da-4ff35c1c7f54\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") "
Oct 29 23:01:26.907853 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.907775    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-tmp-dir\") pod \"3e631530-8524-445b-90da-4ff35c1c7f54\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") "
Oct 29 23:01:26.907853 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.907816    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e631530-8524-445b-90da-4ff35c1c7f54-kube-api-access-krddp\" (UniqueName: \"kubernetes.io/projected/3e631530-8524-445b-90da-4ff35c1c7f54-kube-api-access-krddp\") pod \"3e631530-8524-445b-90da-4ff35c1c7f54\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") "
Oct 29 23:01:26.908021 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.907854    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-dshm\") pod \"3e631530-8524-445b-90da-4ff35c1c7f54\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") "
Oct 29 23:01:26.908021 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.907937    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-model-cache\") pod \"3e631530-8524-445b-90da-4ff35c1c7f54\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") "
Oct 29 23:01:26.908021 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.908014    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-home\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-home\") pod \"3e631530-8524-445b-90da-4ff35c1c7f54\" (UID: \"3e631530-8524-445b-90da-4ff35c1c7f54\") "
Oct 29 23:01:26.908203 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.908073    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-tmp-dir" pod "3e631530-8524-445b-90da-4ff35c1c7f54" (UID: "3e631530-8524-445b-90da-4ff35c1c7f54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:26.908356 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.908288    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-model-cache" pod "3e631530-8524-445b-90da-4ff35c1c7f54" (UID: "3e631530-8524-445b-90da-4ff35c1c7f54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:26.908504 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.908390    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-home" pod "3e631530-8524-445b-90da-4ff35c1c7f54" (UID: "3e631530-8524-445b-90da-4ff35c1c7f54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:26.908766 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.908744    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6gqq\" (UniqueName: \"kubernetes.io/projected/c03c61aa-d816-451a-80f0-c636cf8800a6-kube-api-access-b6gqq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:26.908862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.908772    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:26.908862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.908788    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:26.908862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.908801    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:26.908862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.908816    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c03c61aa-d816-451a-80f0-c636cf8800a6-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:26.908862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.908828    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:26.908862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.908841    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:26.908862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.908854    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c03c61aa-d816-451a-80f0-c636cf8800a6-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:26.910482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.910457    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-dshm" pod "3e631530-8524-445b-90da-4ff35c1c7f54" (UID: "3e631530-8524-445b-90da-4ff35c1c7f54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:26.910654 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.910630    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e631530-8524-445b-90da-4ff35c1c7f54-tls-certs" pod "3e631530-8524-445b-90da-4ff35c1c7f54" (UID: "3e631530-8524-445b-90da-4ff35c1c7f54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:26.911103 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.911075    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e631530-8524-445b-90da-4ff35c1c7f54-kube-api-access-krddp" pod "3e631530-8524-445b-90da-4ff35c1c7f54" (UID: "3e631530-8524-445b-90da-4ff35c1c7f54"). InnerVolumeSpecName "kube-api-access-krddp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:26.960814 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:26.960779    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:27.010109 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.010033    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e631530-8524-445b-90da-4ff35c1c7f54-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.010109 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.010068    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krddp\" (UniqueName: \"kubernetes.io/projected/3e631530-8524-445b-90da-4ff35c1c7f54-kube-api-access-krddp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.010109 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.010082    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e631530-8524-445b-90da-4ff35c1c7f54-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:27.301557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.289508    4860 generic.go:356] "Generic (PLEG): container finished" podID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerID="df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e" exitCode=0
Oct 29 23:01:27.301557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.289739    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"
Oct 29 23:01:27.301557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.290573    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" event={"ID":"c03c61aa-d816-451a-80f0-c636cf8800a6","Type":"ContainerDied","Data":"df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e"}
Oct 29 23:01:27.301557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.290636    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg" event={"ID":"c03c61aa-d816-451a-80f0-c636cf8800a6","Type":"ContainerDied","Data":"7b7bd084034297c601fab789db7109b9877667b7d3368ff0e84592916d38839c"}
Oct 29 23:01:27.301557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.290660    4860 scope.go:122] "RemoveContainer" containerID="df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e"
Oct 29 23:01:27.374464 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.374424    4860 generic.go:356] "Generic (PLEG): container finished" podID="3e631530-8524-445b-90da-4ff35c1c7f54" containerID="fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f" exitCode=0
Oct 29 23:01:27.374631 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.374531    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:27.374705 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.374625    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" event={"ID":"3e631530-8524-445b-90da-4ff35c1c7f54","Type":"ContainerDied","Data":"fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f"}
Oct 29 23:01:27.374705 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.374675    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v" event={"ID":"3e631530-8524-445b-90da-4ff35c1c7f54","Type":"ContainerDied","Data":"d8527fedad6a6497427ebe78b01708d49a2f83e6685c45115538985f1374a033"}
Oct 29 23:01:27.374829 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.374735    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"
Oct 29 23:01:27.419410 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.419388    4860 scope.go:122] "RemoveContainer" containerID="df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e"
Oct 29 23:01:27.419782 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:27.419762    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e\": container with ID starting with df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e not found: ID does not exist" containerID="df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e"
Oct 29 23:01:27.419842 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.419791    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e"} err="failed to get container status \"df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e\": rpc error: code = NotFound desc = could not find container \"df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e\": container with ID starting with df3ccaf8bc602a79d858ae084078e111bcc6041a5ea6a88c0e6f160d391c246e not found: ID does not exist"
Oct 29 23:01:27.419842 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.419808    4860 scope.go:122] "RemoveContainer" containerID="fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f"
Oct 29 23:01:27.430647 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.430625    4860 scope.go:122] "RemoveContainer" containerID="fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f"
Oct 29 23:01:27.430909 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:27.430880    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f\": container with ID starting with fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f not found: ID does not exist" containerID="fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f"
Oct 29 23:01:27.430984 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.430916    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f"} err="failed to get container status \"fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f\": rpc error: code = NotFound desc = could not find container \"fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f\": container with ID starting with fa9e128ac84ea089754bdc1ccf48dcfef1aca1660a0c39f226994d58e857920f not found: ID does not exist"
Oct 29 23:01:27.441584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.441561    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"]
Oct 29 23:01:27.446574 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.446549    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/distinct-w4-08ebee-kserve-65f799894d-jgvpg"]
Oct 29 23:01:27.479463 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.479441    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"]
Oct 29 23:01:27.488219 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.488196    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-08ebee/simulator-w4-08ebee-kserve-6ff65c86bd-dgd5v"]
Oct 29 23:01:27.495893 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.495870    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"]
Oct 29 23:01:27.496514 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496499    4860 cpu_manager.go:414] "containerMap: removing container" podUID="27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" containerName="istio-proxy"
Oct 29 23:01:27.496575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496516    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" containerName="istio-proxy"
Oct 29 23:01:27.496575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496530    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05" containerName="cleanup"
Oct 29 23:01:27.496575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496535    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e276a4c-e944-495d-8d4e-f80ebd436f05" containerName="cleanup"
Oct 29 23:01:27.496575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496544    4860 cpu_manager.go:414] "containerMap: removing container" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main"
Oct 29 23:01:27.496575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496548    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main"
Oct 29 23:01:27.496575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496556    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6" containerName="cleanup"
Oct 29 23:01:27.496575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496562    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6" containerName="cleanup"
Oct 29 23:01:27.496575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496568    4860 cpu_manager.go:414] "containerMap: removing container" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd" containerName="cleanup"
Oct 29 23:01:27.496575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496573    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496582    4860 cpu_manager.go:414] "containerMap: removing container" podUID="32dbf81d-0294-4922-b785-1a0b3950017a" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496587    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32dbf81d-0294-4922-b785-1a0b3950017a" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496596    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1d25d5c6-0825-4c7e-9926-4ab0add384c4" containerName="maas-api"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496601    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d25d5c6-0825-4c7e-9926-4ab0add384c4" containerName="maas-api"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496609    4860 cpu_manager.go:414] "containerMap: removing container" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496613    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496619    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496623    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496631    4860 cpu_manager.go:414] "containerMap: removing container" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496635    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496644    4860 cpu_manager.go:414] "containerMap: removing container" podUID="056308d8-3cce-4df1-a0a9-e0486218e4f5" containerName="payload-processing"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496649    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056308d8-3cce-4df1-a0a9-e0486218e4f5" containerName="payload-processing"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496657    4860 cpu_manager.go:414] "containerMap: removing container" podUID="88824988-8fe2-4a1b-af86-31647faa8af2" containerName="istio-proxy"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496661    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88824988-8fe2-4a1b-af86-31647faa8af2" containerName="istio-proxy"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496672    4860 cpu_manager.go:414] "containerMap: removing container" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496676    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496682    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496687    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496692    4860 cpu_manager.go:414] "containerMap: removing container" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496696    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496706    4860 cpu_manager.go:414] "containerMap: removing container" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496710    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496715    4860 cpu_manager.go:414] "containerMap: removing container" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496720    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496727    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43" containerName="payload-pre-processing"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496732    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43" containerName="payload-pre-processing"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496740    4860 cpu_manager.go:414] "containerMap: removing container" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496744    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496751    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496756    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496761    4860 cpu_manager.go:414] "containerMap: removing container" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496766    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496774    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9e56389b-3fc6-4877-9034-efdf79f2df59" containerName="istio-proxy"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496779    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e56389b-3fc6-4877-9034-efdf79f2df59" containerName="istio-proxy"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496786    4860 cpu_manager.go:414] "containerMap: removing container" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496790    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496802    4860 cpu_manager.go:414] "containerMap: removing container" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496806    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496814    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f91089d4-1a38-4d94-87f8-8131ab078130" containerName="payload-processing"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496818    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91089d4-1a38-4d94-87f8-8131ab078130" containerName="payload-processing"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496824    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496828    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496837    4860 cpu_manager.go:414] "containerMap: removing container" podUID="55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" containerName="istio-proxy"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496841    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" containerName="istio-proxy"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496846    4860 cpu_manager.go:414] "containerMap: removing container" podUID="be999b8c-3f77-451a-9d23-be1db342ce44" containerName="maas-api"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496850    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be999b8c-3f77-451a-9d23-be1db342ce44" containerName="maas-api"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496858    4860 cpu_manager.go:414] "containerMap: removing container" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496862    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" containerName="cleanup"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496871    4860 cpu_manager.go:414] "containerMap: removing container" podUID="335d830d-7cdd-4527-a215-6f247fbf11b8" containerName="payload-pre-processing"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496876    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335d830d-7cdd-4527-a215-6f247fbf11b8" containerName="payload-pre-processing"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496886    4860 cpu_manager.go:414] "containerMap: removing container" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy"
Oct 29 23:01:27.496869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496891    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496899    4860 cpu_manager.go:414] "containerMap: removing container" podUID="9f4a5ad2-3f37-4854-975d-355793e94b71" containerName="payload-pre-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496904    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4a5ad2-3f37-4854-975d-355793e94b71" containerName="payload-pre-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496910    4860 cpu_manager.go:414] "containerMap: removing container" podUID="130135b0-3ddc-47eb-a913-5648e0e93b8d" containerName="payload-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496914    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130135b0-3ddc-47eb-a913-5648e0e93b8d" containerName="payload-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496920    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496924    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496930    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496934    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496941    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ec6cc782-8039-4b69-af59-053bc06853ce" containerName="revoke-keys"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496945    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6cc782-8039-4b69-af59-053bc06853ce" containerName="revoke-keys"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496957    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c7208a84-7581-40bd-8b3e-f8d399feabb0" containerName="payload-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496961    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7208a84-7581-40bd-8b3e-f8d399feabb0" containerName="payload-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496967    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f864ec3f-6650-4081-9656-cb8f9009b4f0" containerName="istio-proxy"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496984    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f864ec3f-6650-4081-9656-cb8f9009b4f0" containerName="istio-proxy"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496991    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2247bdec-f506-4144-a05a-884528c5c132" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.496996    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2247bdec-f506-4144-a05a-884528c5c132" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497004    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497009    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497101    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4a5ad2-3f37-4854-975d-355793e94b71" containerName="payload-pre-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497111    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="130135b0-3ddc-47eb-a913-5648e0e93b8d" containerName="payload-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497119    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" containerName="main"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497126    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6cc782-8039-4b69-af59-053bc06853ce" containerName="revoke-keys"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497132    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7208a84-7581-40bd-8b3e-f8d399feabb0" containerName="payload-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497141    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f864ec3f-6650-4081-9656-cb8f9009b4f0" containerName="istio-proxy"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497148    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f461145-f50b-4ebd-b9fa-25edb7d2af1a" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497155    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497160    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" containerName="istio-proxy"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497166    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e4fb66-49ec-436a-8e2a-8890f83ed081" containerName="main"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497171    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497176    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497180    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d25d5c6-0825-4c7e-9926-4ab0add384c4" containerName="maas-api"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497185    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e206a5-c7f6-4865-98bd-ca3a68f5312a" containerName="main"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497190    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497197    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27bd901-d6a1-48c3-91fe-b9c12b3d7b76" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497204    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497209    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="056308d8-3cce-4df1-a0a9-e0486218e4f5" containerName="payload-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497214    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="88824988-8fe2-4a1b-af86-31647faa8af2" containerName="istio-proxy"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497220    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497225    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497230    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497236    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a131e4-4792-4ab4-98d8-b4cdd6d7ad43" containerName="payload-pre-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497242    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497247    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497252    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e56389b-3fc6-4877-9034-efdf79f2df59" containerName="istio-proxy"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497258    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" containerName="main"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497264    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497269    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91089d4-1a38-4d94-87f8-8131ab078130" containerName="payload-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497274    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd9c0c5-d7b8-4dbd-8c25-2eb4f64e500c" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497280    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e501b2-4c2a-4271-a9f5-5435e6ea3a9c" containerName="istio-proxy"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497285    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="be999b8c-3f77-451a-9d23-be1db342ce44" containerName="maas-api"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497291    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="335d830d-7cdd-4527-a215-6f247fbf11b8" containerName="payload-pre-processing"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497297    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2247bdec-f506-4144-a05a-884528c5c132" containerName="cleanup"
Oct 29 23:01:27.498492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.497301    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc22e15-7269-4307-a9bf-ad39641f7287" containerName="istio-proxy"
Oct 29 23:01:27.502210 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.502193    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"
Oct 29 23:01:27.519874 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.519853    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"]
Oct 29 23:01:27.523565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.523548    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-75b68d\""
Oct 29 23:01:27.523668 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.523575    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-75b68d-dockercfg-lpp4d\""
Oct 29 23:01:27.530911 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.530876    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ae3f75-4562-4785-81ab-3c17b887924d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb\" (UID: \"c6ae3f75-4562-4785-81ab-3c17b887924d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"
Oct 29 23:01:27.531036 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.530975    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfl56\" (UniqueName: \"kubernetes.io/projected/c6ae3f75-4562-4785-81ab-3c17b887924d-kube-api-access-dfl56\") pod \"payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb\" (UID: \"c6ae3f75-4562-4785-81ab-3c17b887924d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"
Oct 29 23:01:27.632272 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.632207    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ae3f75-4562-4785-81ab-3c17b887924d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb\" (UID: \"c6ae3f75-4562-4785-81ab-3c17b887924d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"
Oct 29 23:01:27.632272 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.632262    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfl56\" (UniqueName: \"kubernetes.io/projected/c6ae3f75-4562-4785-81ab-3c17b887924d-kube-api-access-dfl56\") pod \"payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb\" (UID: \"c6ae3f75-4562-4785-81ab-3c17b887924d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"
Oct 29 23:01:27.632938 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.632919    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ae3f75-4562-4785-81ab-3c17b887924d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb\" (UID: \"c6ae3f75-4562-4785-81ab-3c17b887924d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"
Oct 29 23:01:27.656685 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.656656    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfl56\" (UniqueName: \"kubernetes.io/projected/c6ae3f75-4562-4785-81ab-3c17b887924d-kube-api-access-dfl56\") pod \"payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb\" (UID: \"c6ae3f75-4562-4785-81ab-3c17b887924d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"
Oct 29 23:01:27.825904 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.825873    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:27.839066 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.839037    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"
Oct 29 23:01:27.964965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:27.950097    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:28.101155 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:28.100556    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:28.138042 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:28.137996    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27fff8e8-73e3-4a52-8f3b-032ea9abdfcc" path="/var/lib/kubelet/pods/27fff8e8-73e3-4a52-8f3b-032ea9abdfcc/volumes"
Oct 29 23:01:28.167398 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:28.159703    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e631530-8524-445b-90da-4ff35c1c7f54" path="/var/lib/kubelet/pods/3e631530-8524-445b-90da-4ff35c1c7f54/volumes"
Oct 29 23:01:28.187798 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:28.169905    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c03c61aa-d816-451a-80f0-c636cf8800a6" path="/var/lib/kubelet/pods/c03c61aa-d816-451a-80f0-c636cf8800a6/volumes"
Oct 29 23:01:28.187798 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:28.178401    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"]
Oct 29 23:01:28.388271 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:28.388230    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb" event={"ID":"c6ae3f75-4562-4785-81ab-3c17b887924d","Type":"ContainerStarted","Data":"7629fb62d98fa12d39cf0c5432c1c8b1f343e155b6cc5c88ce5b4605e0203ef0"}
Oct 29 23:01:28.388694 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:28.388305    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"
Oct 29 23:01:28.388694 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:28.388343    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb" event={"ID":"c6ae3f75-4562-4785-81ab-3c17b887924d","Type":"ContainerStarted","Data":"6c8796a86d24278967daeb02cc8a403870dfd692d28f9deb403ddb4c94027aaf"}
Oct 29 23:01:28.433510 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:28.433459    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb" podStartSLOduration=1.433446157 podStartE2EDuration="1.433446157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:28.430410379 +0000 UTC m=+2830.455186509" watchObservedRunningTime="2026-10-29 23:01:28.433446157 +0000 UTC m=+2830.458222288"
Oct 29 23:01:28.714731 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:28.714648    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q" podUID="100f773a-93c3-49ef-b704-393f72d7b181" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:29.952612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:29.952505    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:30.076452 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:30.076418    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:30.405242 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:30.405159    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:30.801248 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:30.801213    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:31.475918 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.475885    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"]
Oct 29 23:01:31.476557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.476523    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6" containerName="cleanup"
Oct 29 23:01:31.476557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.476542    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6" containerName="cleanup"
Oct 29 23:01:31.476734 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.476696    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="521dba0b-944b-4ee0-8e00-60fa687c94bd" containerName="cleanup"
Oct 29 23:01:31.476734 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.476708    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b151b922-7435-4d4e-a403-3c911de5d8f6" containerName="cleanup"
Oct 29 23:01:31.476734 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.476719    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7a942f-a12b-42b1-8b11-b6e2c5103798" containerName="cleanup"
Oct 29 23:01:31.476734 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.476726    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8314b5-c567-4ebd-8a0b-d55cf6b3047d" containerName="cleanup"
Oct 29 23:01:31.480537 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.480519    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:31.507702 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.507447    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:01:31.507702 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.507540    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:01:31.514802 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.514770    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcgzr\" (UniqueName: \"kubernetes.io/projected/4fa5e424-881b-483d-a583-5975412c2e06-kube-api-access-zcgzr\") pod \"maas-api-revoke-sub-5478a7af-db96c\" (UID: \"4fa5e424-881b-483d-a583-5975412c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:31.515007 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.514986    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5478a7af-db96c\" (UID: \"4fa5e424-881b-483d-a583-5975412c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:31.515138 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.515121    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5478a7af-db96c\" (UID: \"4fa5e424-881b-483d-a583-5975412c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:31.515412 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.515387    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"]
Oct 29 23:01:31.616011 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.615981    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5478a7af-db96c\" (UID: \"4fa5e424-881b-483d-a583-5975412c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:31.616153 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.616053    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgzr\" (UniqueName: \"kubernetes.io/projected/4fa5e424-881b-483d-a583-5975412c2e06-kube-api-access-zcgzr\") pod \"maas-api-revoke-sub-5478a7af-db96c\" (UID: \"4fa5e424-881b-483d-a583-5975412c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:31.616199 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.616157    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5478a7af-db96c\" (UID: \"4fa5e424-881b-483d-a583-5975412c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:31.616614 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.616595    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5478a7af-db96c\" (UID: \"4fa5e424-881b-483d-a583-5975412c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:31.618432 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.618415    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5478a7af-db96c\" (UID: \"4fa5e424-881b-483d-a583-5975412c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:31.635839 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.635819    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcgzr\" (UniqueName: \"kubernetes.io/projected/4fa5e424-881b-483d-a583-5975412c2e06-kube-api-access-zcgzr\") pod \"maas-api-revoke-sub-5478a7af-db96c\" (UID: \"4fa5e424-881b-483d-a583-5975412c2e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:31.796484 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.796423    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:31.939876 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:31.939832    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:32.081905 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:32.081497    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:32.195360 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:32.195149    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"]
Oct 29 23:01:32.426928 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:32.426843    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c" event={"ID":"4fa5e424-881b-483d-a583-5975412c2e06","Type":"ContainerStarted","Data":"7aeee820d75e99b19add709566e8f330b0e90a2cd6b568b9d7f83bafb4db514c"}
Oct 29 23:01:32.734484 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:32.734448    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:32.868393 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:32.868314    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:33.148480 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.148377    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:33.441364 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.441316    4860 generic.go:356] "Generic (PLEG): container finished" podID="4fa5e424-881b-483d-a583-5975412c2e06" containerID="79e65189dc16243f286400ea0fe09c4259e7b91e638ae8e3ea150baf3025885d" exitCode=0
Oct 29 23:01:33.441510 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.441382    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c" event={"ID":"4fa5e424-881b-483d-a583-5975412c2e06","Type":"ContainerDied","Data":"79e65189dc16243f286400ea0fe09c4259e7b91e638ae8e3ea150baf3025885d"}
Oct 29 23:01:33.685662 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.684502    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:33.716845 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.716818    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"]
Oct 29 23:01:33.721255 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.721239    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:33.740694 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.740668    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"]
Oct 29 23:01:33.751188 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.751161    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8\" (UID: \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:33.751412 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.751373    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6h2\" (UniqueName: \"kubernetes.io/projected/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-kube-api-access-vg6h2\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8\" (UID: \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:33.751694 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.751668    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8\" (UID: \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:33.822108 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.822069    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:33.825177 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.825151    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"]
Oct 29 23:01:33.829535 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.829516    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"
Oct 29 23:01:33.845569 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.845541    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9d6d4e\""
Oct 29 23:01:33.851248 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.851227    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"]
Oct 29 23:01:33.853220 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.853197    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77\" (UID: \"2130a25f-43b2-494d-a9f3-e22e9dbfeb73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"
Oct 29 23:01:33.853309 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.853234    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr8gp\" (UniqueName: \"kubernetes.io/projected/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-kube-api-access-zr8gp\") pod \"payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77\" (UID: \"2130a25f-43b2-494d-a9f3-e22e9dbfeb73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"
Oct 29 23:01:33.853309 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.853276    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8\" (UID: \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:33.853502 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.853477    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8\" (UID: \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:33.853570 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.853523    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6h2\" (UniqueName: \"kubernetes.io/projected/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-kube-api-access-vg6h2\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8\" (UID: \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:33.853933 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.853909    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8\" (UID: \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:33.856003 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.855981    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8\" (UID: \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:33.864522 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.864501    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9d6d4e-dockercfg-7w6tz\""
Oct 29 23:01:33.873758 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:33.873736    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6h2\" (UniqueName: \"kubernetes.io/projected/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-kube-api-access-vg6h2\") pod \"maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8\" (UID: \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:34.033173 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.031866    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"]
Oct 29 23:01:34.033173 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.032543    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77\" (UID: \"2130a25f-43b2-494d-a9f3-e22e9dbfeb73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"
Oct 29 23:01:34.033173 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.032593    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr8gp\" (UniqueName: \"kubernetes.io/projected/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-kube-api-access-zr8gp\") pod \"payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77\" (UID: \"2130a25f-43b2-494d-a9f3-e22e9dbfeb73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"
Oct 29 23:01:34.040388 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.033969    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77\" (UID: \"2130a25f-43b2-494d-a9f3-e22e9dbfeb73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"
Oct 29 23:01:34.041391 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.041356    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:34.056853 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.048491    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"]
Oct 29 23:01:34.056853 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.048608    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"
Oct 29 23:01:34.084680 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.084597    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr8gp\" (UniqueName: \"kubernetes.io/projected/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-kube-api-access-zr8gp\") pod \"payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77\" (UID: \"2130a25f-43b2-494d-a9f3-e22e9dbfeb73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"
Oct 29 23:01:34.147831 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.147801    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"
Oct 29 23:01:34.203152 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.203104    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:34.256290 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.256253    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz46m\" (UniqueName: \"kubernetes.io/projected/e6821c21-38c0-4003-bfe5-66adff17b540-kube-api-access-fz46m\") pod \"payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn\" (UID: \"e6821c21-38c0-4003-bfe5-66adff17b540\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"
Oct 29 23:01:34.260241 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.256461    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6821c21-38c0-4003-bfe5-66adff17b540-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn\" (UID: \"e6821c21-38c0-4003-bfe5-66adff17b540\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"
Oct 29 23:01:34.357838 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.357748    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz46m\" (UniqueName: \"kubernetes.io/projected/e6821c21-38c0-4003-bfe5-66adff17b540-kube-api-access-fz46m\") pod \"payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn\" (UID: \"e6821c21-38c0-4003-bfe5-66adff17b540\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"
Oct 29 23:01:34.357998 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.357861    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6821c21-38c0-4003-bfe5-66adff17b540-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn\" (UID: \"e6821c21-38c0-4003-bfe5-66adff17b540\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"
Oct 29 23:01:34.358858 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.358826    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6821c21-38c0-4003-bfe5-66adff17b540-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn\" (UID: \"e6821c21-38c0-4003-bfe5-66adff17b540\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"
Oct 29 23:01:34.377539 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.377513    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz46m\" (UniqueName: \"kubernetes.io/projected/e6821c21-38c0-4003-bfe5-66adff17b540-kube-api-access-fz46m\") pod \"payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn\" (UID: \"e6821c21-38c0-4003-bfe5-66adff17b540\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"
Oct 29 23:01:34.423718 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:01:34.423690    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa5fef3_7b5a_48c5_abc3_6f21b9e4ec54.slice/crio-79b04985e68ac754b37c1c54309e72805154dfd53c8f07e34a861532f51a0a1d WatchSource:0}: Error finding container 79b04985e68ac754b37c1c54309e72805154dfd53c8f07e34a861532f51a0a1d: Status 404 returned error can't find the container with id 79b04985e68ac754b37c1c54309e72805154dfd53c8f07e34a861532f51a0a1d
Oct 29 23:01:34.424027 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.424008    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"]
Oct 29 23:01:34.440102 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.440081    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"
Oct 29 23:01:34.453107 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.453079    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8" event={"ID":"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54","Type":"ContainerStarted","Data":"79b04985e68ac754b37c1c54309e72805154dfd53c8f07e34a861532f51a0a1d"}
Oct 29 23:01:34.481108 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:01:34.481050    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2130a25f_43b2_494d_a9f3_e22e9dbfeb73.slice/crio-500a93f10ea9764f294e79babeacea374ce36f3763aee5532df52fa7b09ec745 WatchSource:0}: Error finding container 500a93f10ea9764f294e79babeacea374ce36f3763aee5532df52fa7b09ec745: Status 404 returned error can't find the container with id 500a93f10ea9764f294e79babeacea374ce36f3763aee5532df52fa7b09ec745
Oct 29 23:01:34.507103 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.507064    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"]
Oct 29 23:01:34.658026 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.657989    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:34.661566 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.661538    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa5e424-881b-483d-a583-5975412c2e06-kube-api-access-zcgzr\" (UniqueName: \"kubernetes.io/projected/4fa5e424-881b-483d-a583-5975412c2e06-kube-api-access-zcgzr\") pod \"4fa5e424-881b-483d-a583-5975412c2e06\" (UID: \"4fa5e424-881b-483d-a583-5975412c2e06\") "
Oct 29 23:01:34.663796 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.663764    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa5e424-881b-483d-a583-5975412c2e06-kube-api-access-zcgzr" pod "4fa5e424-881b-483d-a583-5975412c2e06" (UID: "4fa5e424-881b-483d-a583-5975412c2e06"). InnerVolumeSpecName "kube-api-access-zcgzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:34.763283 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.763254    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-xks-ca\") pod \"4fa5e424-881b-483d-a583-5975412c2e06\" (UID: \"4fa5e424-881b-483d-a583-5975412c2e06\") "
Oct 29 23:01:34.763745 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.763393    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-openshift-ca\") pod \"4fa5e424-881b-483d-a583-5975412c2e06\" (UID: \"4fa5e424-881b-483d-a583-5975412c2e06\") "
Oct 29 23:01:34.763845 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.763808    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-openshift-ca" pod "4fa5e424-881b-483d-a583-5975412c2e06" (UID: "4fa5e424-881b-483d-a583-5975412c2e06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:34.763912 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.763894    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcgzr\" (UniqueName: \"kubernetes.io/projected/4fa5e424-881b-483d-a583-5975412c2e06-kube-api-access-zcgzr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:34.765227 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:01:34.765199    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6821c21_38c0_4003_bfe5_66adff17b540.slice/crio-c8f34fa3f6eaf22562b0f094cddbbd417aa70e390991eb933ec0d47d16a6a3c2 WatchSource:0}: Error finding container c8f34fa3f6eaf22562b0f094cddbbd417aa70e390991eb933ec0d47d16a6a3c2: Status 404 returned error can't find the container with id c8f34fa3f6eaf22562b0f094cddbbd417aa70e390991eb933ec0d47d16a6a3c2
Oct 29 23:01:34.765417 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.765398    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-xks-ca" pod "4fa5e424-881b-483d-a583-5975412c2e06" (UID: "4fa5e424-881b-483d-a583-5975412c2e06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:34.767344 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.767308    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"]
Oct 29 23:01:34.865508 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.865482    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:34.865508 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:34.865507    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa5e424-881b-483d-a583-5975412c2e06-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:35.462783 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.462753    4860 generic.go:356] "Generic (PLEG): container finished" podID="5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54" containerID="5b1a882fd1a5472f33f89ac578b8eb1d8885e9be46af65b2d2e2e9373ad5f2cf" exitCode=0
Oct 29 23:01:35.462938 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.462816    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8" event={"ID":"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54","Type":"ContainerDied","Data":"5b1a882fd1a5472f33f89ac578b8eb1d8885e9be46af65b2d2e2e9373ad5f2cf"}
Oct 29 23:01:35.464597 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.464571    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77" event={"ID":"2130a25f-43b2-494d-a9f3-e22e9dbfeb73","Type":"ContainerStarted","Data":"0516c48707fbc5022ae6416bc10c6ed978d9ae94dd652e30a9b6dec47b7ccf06"}
Oct 29 23:01:35.464715 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.464617    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"
Oct 29 23:01:35.464715 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.464633    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77" event={"ID":"2130a25f-43b2-494d-a9f3-e22e9dbfeb73","Type":"ContainerStarted","Data":"500a93f10ea9764f294e79babeacea374ce36f3763aee5532df52fa7b09ec745"}
Oct 29 23:01:35.466379 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.466352    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn" event={"ID":"e6821c21-38c0-4003-bfe5-66adff17b540","Type":"ContainerStarted","Data":"dd5f9cbf9d0bfc793df3c25f9021ec3bfee05c6953738be9e572c629b01ae06e"}
Oct 29 23:01:35.466464 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.466402    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn" event={"ID":"e6821c21-38c0-4003-bfe5-66adff17b540","Type":"ContainerStarted","Data":"c8f34fa3f6eaf22562b0f094cddbbd417aa70e390991eb933ec0d47d16a6a3c2"}
Oct 29 23:01:35.466464 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.466430    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"
Oct 29 23:01:35.467834 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.467820    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"
Oct 29 23:01:35.467939 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.467817    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c" event={"ID":"4fa5e424-881b-483d-a583-5975412c2e06","Type":"ContainerDied","Data":"7aeee820d75e99b19add709566e8f330b0e90a2cd6b568b9d7f83bafb4db514c"}
Oct 29 23:01:35.468012 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.467971    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aeee820d75e99b19add709566e8f330b0e90a2cd6b568b9d7f83bafb4db514c"
Oct 29 23:01:35.556142 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.556060    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77" podStartSLOduration=2.556049764 podStartE2EDuration="2.556049764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:35.555879458 +0000 UTC m=+2837.580655607" watchObservedRunningTime="2026-10-29 23:01:35.556049764 +0000 UTC m=+2837.580825895"
Oct 29 23:01:35.599828 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:35.599784    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn" podStartSLOduration=2.599772894 podStartE2EDuration="2.599772894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:35.59635141 +0000 UTC m=+2837.621127561" watchObservedRunningTime="2026-10-29 23:01:35.599772894 +0000 UTC m=+2837.624549025"
Oct 29 23:01:36.276746 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:36.276717    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:36.640046 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:36.640021    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:36.687661 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:36.687629    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-xks-ca\") pod \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\" (UID: \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\") "
Oct 29 23:01:36.687982 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:36.687937    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-kube-api-access-vg6h2\" (UniqueName: \"kubernetes.io/projected/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-kube-api-access-vg6h2\") pod \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\" (UID: \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\") "
Oct 29 23:01:36.688091 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:36.688063    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-openshift-ca\") pod \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\" (UID: \"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54\") "
Oct 29 23:01:36.688828 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:36.688796    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-openshift-ca" pod "5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54" (UID: "5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:36.690486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:36.690451    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-kube-api-access-vg6h2" pod "5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54" (UID: "5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54"). InnerVolumeSpecName "kube-api-access-vg6h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:36.690615 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:36.690600    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-xks-ca" pod "5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54" (UID: "5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:36.697982 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:36.697951    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:36.698105 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:36.697985    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg6h2\" (UniqueName: \"kubernetes.io/projected/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-kube-api-access-vg6h2\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:36.698105 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:36.698001    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:37.492832 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:37.492794    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8" event={"ID":"5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54","Type":"ContainerDied","Data":"79b04985e68ac754b37c1c54309e72805154dfd53c8f07e34a861532f51a0a1d"}
Oct 29 23:01:37.492832 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:37.492819    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"
Oct 29 23:01:37.493353 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:37.492867    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79b04985e68ac754b37c1c54309e72805154dfd53c8f07e34a861532f51a0a1d"
Oct 29 23:01:38.206180 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:38.206146    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:38.714788 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:38.714746    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q" podUID="100f773a-93c3-49ef-b704-393f72d7b181" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:39.402044 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:39.402017    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"
Oct 29 23:01:39.603561 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:39.603526    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:39.774528 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:39.774499    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:40.832927 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:40.832892    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:42.454957 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.454897    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"
Oct 29 23:01:42.481031 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.481006    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:42.554677 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.554643    4860 generic.go:356] "Generic (PLEG): container finished" podID="100f773a-93c3-49ef-b704-393f72d7b181" containerID="8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931" exitCode=0
Oct 29 23:01:42.554841 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.554690    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"
Oct 29 23:01:42.554841 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.554727    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q" event={"ID":"100f773a-93c3-49ef-b704-393f72d7b181","Type":"ContainerDied","Data":"8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931"}
Oct 29 23:01:42.554841 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.554785    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q" event={"ID":"100f773a-93c3-49ef-b704-393f72d7b181","Type":"ContainerDied","Data":"e377b6a7d7b2804d560a659de6dc5c95a7538d6064bba22534dec494de00daf3"}
Oct 29 23:01:42.554841 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.554813    4860 scope.go:122] "RemoveContainer" containerID="8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931"
Oct 29 23:01:42.566625 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.566601    4860 scope.go:122] "RemoveContainer" containerID="8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931"
Oct 29 23:01:42.567014 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:42.566993    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931\": container with ID starting with 8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931 not found: ID does not exist" containerID="8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931"
Oct 29 23:01:42.567112 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.567022    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931"} err="failed to get container status \"8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931\": rpc error: code = NotFound desc = could not find container \"8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931\": container with ID starting with 8574ab9f59433ea76471f034fd963d15fa16eabc08ef45d0306e9a1c18316931 not found: ID does not exist"
Oct 29 23:01:42.633524 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.633501    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/100f773a-93c3-49ef-b704-393f72d7b181-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100f773a-93c3-49ef-b704-393f72d7b181-plugins-config-volume\") pod \"100f773a-93c3-49ef-b704-393f72d7b181\" (UID: \"100f773a-93c3-49ef-b704-393f72d7b181\") "
Oct 29 23:01:42.633730 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.633714    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100f773a-93c3-49ef-b704-393f72d7b181-kube-api-access-tvbnq\" (UniqueName: \"kubernetes.io/projected/100f773a-93c3-49ef-b704-393f72d7b181-kube-api-access-tvbnq\") pod \"100f773a-93c3-49ef-b704-393f72d7b181\" (UID: \"100f773a-93c3-49ef-b704-393f72d7b181\") "
Oct 29 23:01:42.633879 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.633859    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100f773a-93c3-49ef-b704-393f72d7b181-plugins-config-volume" pod "100f773a-93c3-49ef-b704-393f72d7b181" (UID: "100f773a-93c3-49ef-b704-393f72d7b181"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:42.634294 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.634274    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100f773a-93c3-49ef-b704-393f72d7b181-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:42.635792 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.635770    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100f773a-93c3-49ef-b704-393f72d7b181-kube-api-access-tvbnq" pod "100f773a-93c3-49ef-b704-393f72d7b181" (UID: "100f773a-93c3-49ef-b704-393f72d7b181"). InnerVolumeSpecName "kube-api-access-tvbnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:42.735373 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.735303    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvbnq\" (UniqueName: \"kubernetes.io/projected/100f773a-93c3-49ef-b704-393f72d7b181-kube-api-access-tvbnq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:42.910751 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.910718    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"]
Oct 29 23:01:42.915227 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:42.915195    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-08ebee-5f548676d4-st67q"]
Oct 29 23:01:44.115029 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:44.114997    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="100f773a-93c3-49ef-b704-393f72d7b181" path="/var/lib/kubelet/pods/100f773a-93c3-49ef-b704-393f72d7b181/volumes"
Oct 29 23:01:44.118190 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:44.118154    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:46.479438 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:46.479406    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"
Oct 29 23:01:46.479846 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:46.479673    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"
Oct 29 23:01:46.499067 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:46.499039    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:48.010354 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:48.010292    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:48.139732 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:48.139694    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:01:48.269193 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:48.269113    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:48.473962 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:48.473926    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:48.933254 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:48.933220    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:49.994949 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:49.994914    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:50.808313 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:50.808281    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:52.502413 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:52.502379    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:54.064764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:54.064729    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:55.031806 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:55.031776    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"]
Oct 29 23:01:55.043713 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:55.043687    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e823990-ptdgp"]
Oct 29 23:01:55.704109 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:55.704079    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:55.819263 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:55.819231    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:55.966821 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:55.966754    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:56.114049 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:56.114017    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5689700a-6385-4b1a-beda-af609bf6cec4" path="/var/lib/kubelet/pods/5689700a-6385-4b1a-beda-af609bf6cec4/volumes"
Oct 29 23:01:57.339907 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:57.339873    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:57.888839 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:57.888800    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:58.024606 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:58.024577    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:58.462138 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:58.462105    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:59.370261 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.370225    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"]
Oct 29 23:01:59.370541 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.370513    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv" podUID="2a4ac286-93cf-4aaf-a3da-80c5f28417bb" containerName="maas-api" containerID="cri-o://60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048" gracePeriod=30
Oct 29 23:01:59.471063 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.471029    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:59.577912 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.577879    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"]
Oct 29 23:01:59.578192 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.578127    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb" podUID="c6ae3f75-4562-4785-81ab-3c17b887924d" containerName="payload-pre-processing" containerID="cri-o://7629fb62d98fa12d39cf0c5432c1c8b1f343e155b6cc5c88ce5b4605e0203ef0" gracePeriod=30
Oct 29 23:01:59.654633 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.654563    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:59.716184 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.716152    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:01:59.738717 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.738689    4860 generic.go:356] "Generic (PLEG): container finished" podID="2a4ac286-93cf-4aaf-a3da-80c5f28417bb" containerID="60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048" exitCode=0
Oct 29 23:01:59.738953 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.738936    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"
Oct 29 23:01:59.739143 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.739125    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv" event={"ID":"2a4ac286-93cf-4aaf-a3da-80c5f28417bb","Type":"ContainerDied","Data":"60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048"}
Oct 29 23:01:59.739222 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.739172    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv" event={"ID":"2a4ac286-93cf-4aaf-a3da-80c5f28417bb","Type":"ContainerDied","Data":"11d9338fb554e66d9223d6831aad316bb83be371656ab953f5cf6939a5f28106"}
Oct 29 23:01:59.739222 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.739205    4860 scope.go:122] "RemoveContainer" containerID="60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048"
Oct 29 23:01:59.740986 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.740959    4860 generic.go:356] "Generic (PLEG): container finished" podID="c6ae3f75-4562-4785-81ab-3c17b887924d" containerID="7629fb62d98fa12d39cf0c5432c1c8b1f343e155b6cc5c88ce5b4605e0203ef0" exitCode=0
Oct 29 23:01:59.741066 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.741025    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb" event={"ID":"c6ae3f75-4562-4785-81ab-3c17b887924d","Type":"ContainerDied","Data":"7629fb62d98fa12d39cf0c5432c1c8b1f343e155b6cc5c88ce5b4605e0203ef0"}
Oct 29 23:01:59.753745 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.753722    4860 scope.go:122] "RemoveContainer" containerID="60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048"
Oct 29 23:01:59.754039 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:01:59.754013    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048\": container with ID starting with 60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048 not found: ID does not exist" containerID="60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048"
Oct 29 23:01:59.754127 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.754046    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048"} err="failed to get container status \"60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048\": rpc error: code = NotFound desc = could not find container \"60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048\": container with ID starting with 60e541a5bb93ebb7b7763863b19e2fd197092f645788ec944dd479ba06cb5048 not found: ID does not exist"
Oct 29 23:01:59.776287 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.776248    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:59.808382 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.808350    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-openshift-service-ca\") pod \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") "
Oct 29 23:01:59.808539 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.808426    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-metrics-certs\") pod \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") "
Oct 29 23:01:59.808664 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.808644    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-kube-api-access-624rl\" (UniqueName: \"kubernetes.io/projected/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-kube-api-access-624rl\") pod \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") "
Oct 29 23:01:59.808756 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.808735    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-maas-api-tls\") pod \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\" (UID: \"2a4ac286-93cf-4aaf-a3da-80c5f28417bb\") "
Oct 29 23:01:59.809568 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.809533    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-openshift-service-ca" pod "2a4ac286-93cf-4aaf-a3da-80c5f28417bb" (UID: "2a4ac286-93cf-4aaf-a3da-80c5f28417bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:59.812079 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.812052    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-metrics-certs" pod "2a4ac286-93cf-4aaf-a3da-80c5f28417bb" (UID: "2a4ac286-93cf-4aaf-a3da-80c5f28417bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:59.812936 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.812910    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-maas-api-tls" pod "2a4ac286-93cf-4aaf-a3da-80c5f28417bb" (UID: "2a4ac286-93cf-4aaf-a3da-80c5f28417bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:59.813431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.813311    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-kube-api-access-624rl" pod "2a4ac286-93cf-4aaf-a3da-80c5f28417bb" (UID: "2a4ac286-93cf-4aaf-a3da-80c5f28417bb"). InnerVolumeSpecName "kube-api-access-624rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:59.864975 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.864942    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"
Oct 29 23:01:59.945914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.939946    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ae3f75-4562-4785-81ab-3c17b887924d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ae3f75-4562-4785-81ab-3c17b887924d-plugins-config-volume\") pod \"c6ae3f75-4562-4785-81ab-3c17b887924d\" (UID: \"c6ae3f75-4562-4785-81ab-3c17b887924d\") "
Oct 29 23:01:59.945914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.940074    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ae3f75-4562-4785-81ab-3c17b887924d-kube-api-access-dfl56\" (UniqueName: \"kubernetes.io/projected/c6ae3f75-4562-4785-81ab-3c17b887924d-kube-api-access-dfl56\") pod \"c6ae3f75-4562-4785-81ab-3c17b887924d\" (UID: \"c6ae3f75-4562-4785-81ab-3c17b887924d\") "
Oct 29 23:01:59.945914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.940838    4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-openshift-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:59.945914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.940867    4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-metrics-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:59.945914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.940885    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-624rl\" (UniqueName: \"kubernetes.io/projected/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-kube-api-access-624rl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:59.945914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.940897    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a4ac286-93cf-4aaf-a3da-80c5f28417bb-maas-api-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:01:59.945914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.945547    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ae3f75-4562-4785-81ab-3c17b887924d-plugins-config-volume" pod "c6ae3f75-4562-4785-81ab-3c17b887924d" (UID: "c6ae3f75-4562-4785-81ab-3c17b887924d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:59.958364 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:01:59.958318    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ae3f75-4562-4785-81ab-3c17b887924d-kube-api-access-dfl56" pod "c6ae3f75-4562-4785-81ab-3c17b887924d" (UID: "c6ae3f75-4562-4785-81ab-3c17b887924d"). InnerVolumeSpecName "kube-api-access-dfl56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:00.024413 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:00.017053    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:00.043409 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:00.042619    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6ae3f75-4562-4785-81ab-3c17b887924d-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:00.043409 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:00.042653    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfl56\" (UniqueName: \"kubernetes.io/projected/c6ae3f75-4562-4785-81ab-3c17b887924d-kube-api-access-dfl56\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:00.102028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:00.101993    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"]
Oct 29 23:02:00.116130 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:00.116103    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75b68d-7864cfdfcd-z7fwv"]
Oct 29 23:02:00.140249 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:00.140205    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:00.672028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:00.671949    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:00.751002 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:00.750975    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"
Oct 29 23:02:00.751002 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:00.751004    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb" event={"ID":"c6ae3f75-4562-4785-81ab-3c17b887924d","Type":"ContainerDied","Data":"6c8796a86d24278967daeb02cc8a403870dfd692d28f9deb403ddb4c94027aaf"}
Oct 29 23:02:00.751255 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:00.751058    4860 scope.go:122] "RemoveContainer" containerID="7629fb62d98fa12d39cf0c5432c1c8b1f343e155b6cc5c88ce5b4605e0203ef0"
Oct 29 23:02:00.807959 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:00.807928    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"]
Oct 29 23:02:00.819904 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:00.819866    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75b68d-59747b7fb7-7n6zb"]
Oct 29 23:02:02.057985 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:02.057876    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:02.122553 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:02.122516    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a4ac286-93cf-4aaf-a3da-80c5f28417bb" path="/var/lib/kubelet/pods/2a4ac286-93cf-4aaf-a3da-80c5f28417bb/volumes"
Oct 29 23:02:02.123102 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:02.123085    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ae3f75-4562-4785-81ab-3c17b887924d" path="/var/lib/kubelet/pods/c6ae3f75-4562-4785-81ab-3c17b887924d/volumes"
Oct 29 23:02:03.962667 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:03.962635    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:04.084357 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:04.083295    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:04.783862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:04.783825    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:05.028680 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:05.028649    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:05.169963 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:05.169884    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:05.488309 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:05.488272    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:06.022777 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:06.022721    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:06.042382 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:06.042353    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"]
Oct 29 23:02:06.058166 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:06.058138    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e23de74-w9ps5"]
Oct 29 23:02:06.115117 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:06.115091    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="742c9aa7-4d6a-4200-b408-66b7c8f09d7f" path="/var/lib/kubelet/pods/742c9aa7-4d6a-4200-b408-66b7c8f09d7f/volumes"
Oct 29 23:02:06.143207 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:06.143065    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:06.605340 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:06.605291    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:07.236751 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:07.236720    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:07.380864 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:07.380821    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:07.578507 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:07.578425    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:08.002039 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:08.001986    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:08.545906 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:08.545876    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:09.755306 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:09.755268    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:11.056112 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.056080    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"]
Oct 29 23:02:11.056614 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.056354    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x" podUID="1133119f-4582-4d64-9036-b4ebdae3d993" containerName="maas-api" containerID="cri-o://3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0" gracePeriod=30
Oct 29 23:02:11.141362 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.141288    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:11.325969 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.325896    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:11.363470 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.363441    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:02:11.443505 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.443471    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:11.540072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.537959    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-metrics-certs\") pod \"1133119f-4582-4d64-9036-b4ebdae3d993\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") "
Oct 29 23:02:11.540072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.538244    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1133119f-4582-4d64-9036-b4ebdae3d993-kube-api-access-ck99g\" (UniqueName: \"kubernetes.io/projected/1133119f-4582-4d64-9036-b4ebdae3d993-kube-api-access-ck99g\") pod \"1133119f-4582-4d64-9036-b4ebdae3d993\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") "
Oct 29 23:02:11.540072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.538292    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-maas-api-tls\") pod \"1133119f-4582-4d64-9036-b4ebdae3d993\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") "
Oct 29 23:02:11.540072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.538352    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1133119f-4582-4d64-9036-b4ebdae3d993-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1133119f-4582-4d64-9036-b4ebdae3d993-openshift-service-ca\") pod \"1133119f-4582-4d64-9036-b4ebdae3d993\" (UID: \"1133119f-4582-4d64-9036-b4ebdae3d993\") "
Oct 29 23:02:11.540982 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.540949    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1133119f-4582-4d64-9036-b4ebdae3d993-openshift-service-ca" pod "1133119f-4582-4d64-9036-b4ebdae3d993" (UID: "1133119f-4582-4d64-9036-b4ebdae3d993"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:11.559780 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.559747    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-metrics-certs" pod "1133119f-4582-4d64-9036-b4ebdae3d993" (UID: "1133119f-4582-4d64-9036-b4ebdae3d993"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:11.606510 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.606466    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-maas-api-tls" pod "1133119f-4582-4d64-9036-b4ebdae3d993" (UID: "1133119f-4582-4d64-9036-b4ebdae3d993"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:11.614386 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.612522    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1133119f-4582-4d64-9036-b4ebdae3d993-kube-api-access-ck99g" pod "1133119f-4582-4d64-9036-b4ebdae3d993" (UID: "1133119f-4582-4d64-9036-b4ebdae3d993"). InnerVolumeSpecName "kube-api-access-ck99g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:11.646973 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.641963    4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-metrics-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:11.646973 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.642005    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck99g\" (UniqueName: \"kubernetes.io/projected/1133119f-4582-4d64-9036-b4ebdae3d993-kube-api-access-ck99g\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:11.646973 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.642019    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1133119f-4582-4d64-9036-b4ebdae3d993-maas-api-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:11.646973 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.642032    4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1133119f-4582-4d64-9036-b4ebdae3d993-openshift-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:11.671100 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.662585    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:11.779586 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.779553    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"]
Oct 29 23:02:11.787746 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.787711    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-63e89c-gxsh8"]
Oct 29 23:02:11.789628 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.789594    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:11.860270 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.860171    4860 generic.go:356] "Generic (PLEG): container finished" podID="1133119f-4582-4d64-9036-b4ebdae3d993" containerID="3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0" exitCode=0
Oct 29 23:02:11.860270 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.860258    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"
Oct 29 23:02:11.860840 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.860269    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x" event={"ID":"1133119f-4582-4d64-9036-b4ebdae3d993","Type":"ContainerDied","Data":"3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0"}
Oct 29 23:02:11.860840 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.860412    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x" event={"ID":"1133119f-4582-4d64-9036-b4ebdae3d993","Type":"ContainerDied","Data":"5b46e7f1f98b0ce71ee661b7455968959dcd4dac7edfcf4d03b494e6441d00a8"}
Oct 29 23:02:11.860840 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.860452    4860 scope.go:122] "RemoveContainer" containerID="3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0"
Oct 29 23:02:11.878500 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.878477    4860 scope.go:122] "RemoveContainer" containerID="3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0"
Oct 29 23:02:11.878866 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:02:11.878843    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0\": container with ID starting with 3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0 not found: ID does not exist" containerID="3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0"
Oct 29 23:02:11.878972 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.878874    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0"} err="failed to get container status \"3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0\": rpc error: code = NotFound desc = could not find container \"3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0\": container with ID starting with 3e020c4fd637227019402c47c754a5a9746fb7b85768cc18e26ee6a1a9e0ddf0 not found: ID does not exist"
Oct 29 23:02:11.926194 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.926166    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"]
Oct 29 23:02:11.930518 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:11.930493    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e89c-7d66596948-8fw9x"]
Oct 29 23:02:12.114871 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:12.114808    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1133119f-4582-4d64-9036-b4ebdae3d993" path="/var/lib/kubelet/pods/1133119f-4582-4d64-9036-b4ebdae3d993/volumes"
Oct 29 23:02:12.115288 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:12.115273    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54" path="/var/lib/kubelet/pods/5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54/volumes"
Oct 29 23:02:12.284147 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:12.284114    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:13.028275 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:13.028243    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:13.155635 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:13.155599    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:13.283111 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:13.283035    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"]
Oct 29 23:02:13.283304 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:13.283259    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy" containerID="cri-o://92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0" gracePeriod=30
Oct 29 23:02:13.334522 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:13.334489    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:13.754475 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:13.754443    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:14.329335 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:14.329301    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:14.446116 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:14.446081    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:14.601730 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:14.601647    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"]
Oct 29 23:02:14.601939 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:14.601912    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy" containerID="cri-o://28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173" gracePeriod=30
Oct 29 23:02:14.666358 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:14.666293    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:15.202285 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:15.202245    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:15.471790 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:15.471710    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:17.875314 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:17.875270    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:18.060940 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:02:18.060895    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:02:18.552878 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.552715    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:18.636887 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.636859    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 23:02:18.670722 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.669829    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:18.781344 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.781288    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-certs\") pod \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") "
Oct 29 23:02:18.781517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.781371    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-credential-socket\") pod \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") "
Oct 29 23:02:18.781517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.781416    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-data\") pod \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") "
Oct 29 23:02:18.781517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.781461    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-envoy\") pod \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") "
Oct 29 23:02:18.781517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.781513    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-kube-api-access-489hs\" (UniqueName: \"kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-kube-api-access-489hs\") pod \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") "
Oct 29 23:02:18.781765 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.781602    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-certs" pod "ec041e3f-8c83-4807-94f2-40a0636a4dd9" (UID: "ec041e3f-8c83-4807-94f2-40a0636a4dd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:18.781765 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.781606    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-podinfo\") pod \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") "
Oct 29 23:02:18.781765 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.781706    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-credential-socket" pod "ec041e3f-8c83-4807-94f2-40a0636a4dd9" (UID: "ec041e3f-8c83-4807-94f2-40a0636a4dd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:18.781914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.781793    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-data" pod "ec041e3f-8c83-4807-94f2-40a0636a4dd9" (UID: "ec041e3f-8c83-4807-94f2-40a0636a4dd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:18.781914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.781847    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-socket\") pod \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") "
Oct 29 23:02:18.781914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.781900    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-token\" (UniqueName: \"kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-token\") pod \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") "
Oct 29 23:02:18.782088 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.781967    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istiod-ca-cert\") pod \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\" (UID: \"ec041e3f-8c83-4807-94f2-40a0636a4dd9\") "
Oct 29 23:02:18.782151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.782104    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-socket" pod "ec041e3f-8c83-4807-94f2-40a0636a4dd9" (UID: "ec041e3f-8c83-4807-94f2-40a0636a4dd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:18.782478 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.782449    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istiod-ca-cert" pod "ec041e3f-8c83-4807-94f2-40a0636a4dd9" (UID: "ec041e3f-8c83-4807-94f2-40a0636a4dd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:18.782621 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.782559    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.782621 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.782581    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.782621 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.782594    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.782621 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.782609    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.782621 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.782621    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.783903 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.783872    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-podinfo" pod "ec041e3f-8c83-4807-94f2-40a0636a4dd9" (UID: "ec041e3f-8c83-4807-94f2-40a0636a4dd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:18.784031 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.783937    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-kube-api-access-489hs" pod "ec041e3f-8c83-4807-94f2-40a0636a4dd9" (UID: "ec041e3f-8c83-4807-94f2-40a0636a4dd9"). InnerVolumeSpecName "kube-api-access-489hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:18.784031 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.783995    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-envoy" pod "ec041e3f-8c83-4807-94f2-40a0636a4dd9" (UID: "ec041e3f-8c83-4807-94f2-40a0636a4dd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:18.784197 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.784173    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-token" pod "ec041e3f-8c83-4807-94f2-40a0636a4dd9" (UID: "ec041e3f-8c83-4807-94f2-40a0636a4dd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:18.862139 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.861095    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:18.883826 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.883799    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-489hs\" (UniqueName: \"kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-kube-api-access-489hs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.883826 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.883827    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.884267 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.883837    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.884267 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.883845    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec041e3f-8c83-4807-94f2-40a0636a4dd9-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:18.939964 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.939934    4860 generic.go:356] "Generic (PLEG): container finished" podID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerID="92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0" exitCode=0
Oct 29 23:02:18.940146 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.939985    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"
Oct 29 23:02:18.940146 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.940016    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" event={"ID":"ec041e3f-8c83-4807-94f2-40a0636a4dd9","Type":"ContainerDied","Data":"92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0"}
Oct 29 23:02:18.940146 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.940082    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4" event={"ID":"ec041e3f-8c83-4807-94f2-40a0636a4dd9","Type":"ContainerDied","Data":"e181cb13b63897dfbe1131ce26d6fb5a8f37b9876a77c6c0f822d98caaf70e6b"}
Oct 29 23:02:18.940146 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.940114    4860 scope.go:122] "RemoveContainer" containerID="92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0"
Oct 29 23:02:18.954839 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.954821    4860 scope.go:122] "RemoveContainer" containerID="59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00"
Oct 29 23:02:18.965230 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.965207    4860 scope.go:122] "RemoveContainer" containerID="92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0"
Oct 29 23:02:18.965503 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:02:18.965484    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0\": container with ID starting with 92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0 not found: ID does not exist" containerID="92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0"
Oct 29 23:02:18.965557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.965510    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0"} err="failed to get container status \"92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0\": rpc error: code = NotFound desc = could not find container \"92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0\": container with ID starting with 92c9613cfc46158133a86df953939ca05a2bd213d5325cd4604f43cbce1b93c0 not found: ID does not exist"
Oct 29 23:02:18.965557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.965527    4860 scope.go:122] "RemoveContainer" containerID="59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00"
Oct 29 23:02:18.965752 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:02:18.965733    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00\": container with ID starting with 59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00 not found: ID does not exist" containerID="59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00"
Oct 29 23:02:18.965806 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.965761    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00"} err="failed to get container status \"59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00\": rpc error: code = NotFound desc = could not find container \"59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00\": container with ID starting with 59c9baf98287fce6775fefece1579598f0f11c1e9f51f06a6a3201d644621a00 not found: ID does not exist"
Oct 29 23:02:18.976399 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.976377    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"]
Oct 29 23:02:18.976564 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:18.976548    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerName="istio-proxy" containerID="cri-o://61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067" gracePeriod=30
Oct 29 23:02:19.006545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.006523    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"]
Oct 29 23:02:19.010447 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.010241    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-08ebee-openshift-default-5f5bf46967-t8fq4"]
Oct 29 23:02:19.051388 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.051363    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:19.877484 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.877427    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 23:02:19.924667 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.924637    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-certs\") pod \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") "
Oct 29 23:02:19.925062 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.924705    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-envoy\") pod \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") "
Oct 29 23:02:19.925062 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.924770    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-data\") pod \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") "
Oct 29 23:02:19.925062 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.924823    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istiod-ca-cert\") pod \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") "
Oct 29 23:02:19.925062 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.924860    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-socket\") pod \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") "
Oct 29 23:02:19.925062 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.924934    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-certs" pod "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" (UID: "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:19.925062 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925020    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-kube-api-access-84c8v\" (UniqueName: \"kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-kube-api-access-84c8v\") pod \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") "
Oct 29 23:02:19.925062 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925057    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-data" pod "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" (UID: "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:19.925513 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925062    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-token\" (UniqueName: \"kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-token\") pod \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") "
Oct 29 23:02:19.925513 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925159    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-credential-socket\") pod \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") "
Oct 29 23:02:19.925513 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925188    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-socket" pod "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" (UID: "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:19.925513 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925211    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-podinfo\") pod \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\" (UID: \"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f\") "
Oct 29 23:02:19.925513 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925249    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istiod-ca-cert" pod "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" (UID: "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:19.925811 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925534    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-credential-socket" pod "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" (UID: "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:19.925908 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925886    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:19.925965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925917    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:19.925965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925933    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:19.925965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925946    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:19.925965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.925961    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:19.927203 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.927182    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-envoy" pod "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" (UID: "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:19.927784 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.927761    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-token" pod "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" (UID: "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:19.927876 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.927795    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-kube-api-access-84c8v" pod "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" (UID: "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f"). InnerVolumeSpecName "kube-api-access-84c8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:19.928059 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.928038    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-podinfo" pod "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" (UID: "6f45ec0f-a12c-4de7-a655-51ba0d0cd85f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:19.955409 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.955373    4860 generic.go:356] "Generic (PLEG): container finished" podID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerID="28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173" exitCode=0
Oct 29 23:02:19.955615 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.955437    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"
Oct 29 23:02:19.955615 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.955553    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" event={"ID":"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f","Type":"ContainerDied","Data":"28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173"}
Oct 29 23:02:19.955615 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.955609    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h" event={"ID":"6f45ec0f-a12c-4de7-a655-51ba0d0cd85f","Type":"ContainerDied","Data":"ce58061bad85e23ce0e72ee8229af99f9f7e35f52e2b63bd74f2fe930dcfcc49"}
Oct 29 23:02:19.955796 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.955641    4860 scope.go:122] "RemoveContainer" containerID="28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173"
Oct 29 23:02:19.970023 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.970001    4860 scope.go:122] "RemoveContainer" containerID="1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0"
Oct 29 23:02:19.981414 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.981387    4860 scope.go:122] "RemoveContainer" containerID="28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173"
Oct 29 23:02:19.981685 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:02:19.981664    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173\": container with ID starting with 28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173 not found: ID does not exist" containerID="28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173"
Oct 29 23:02:19.981778 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.981699    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173"} err="failed to get container status \"28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173\": rpc error: code = NotFound desc = could not find container \"28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173\": container with ID starting with 28916668a4e95e85d71efa420ebb5d438aa74362c4c3b86d64ec77e0f83a7173 not found: ID does not exist"
Oct 29 23:02:19.981778 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.981720    4860 scope.go:122] "RemoveContainer" containerID="1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0"
Oct 29 23:02:19.982012 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:02:19.981990    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0\": container with ID starting with 1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0 not found: ID does not exist" containerID="1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0"
Oct 29 23:02:19.982105 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:19.982020    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0"} err="failed to get container status \"1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0\": rpc error: code = NotFound desc = could not find container \"1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0\": container with ID starting with 1e4766b6cbbcd3e7e98c826997c98679c3812a9f1861c57af4888ff057b9a9c0 not found: ID does not exist"
Oct 29 23:02:20.013100 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:20.013070    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"]
Oct 29 23:02:20.017081 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:20.017056    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7be71d-openshift-default-8dd8b5bdc-4hx2h"]
Oct 29 23:02:20.027743 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:20.027719    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:20.027743 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:20.027744    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84c8v\" (UniqueName: \"kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-kube-api-access-84c8v\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:20.027875 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:20.027754    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:20.027875 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:20.027764    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:20.119363 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:20.115687    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" path="/var/lib/kubelet/pods/6f45ec0f-a12c-4de7-a655-51ba0d0cd85f/volumes"
Oct 29 23:02:20.119363 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:20.116541    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" path="/var/lib/kubelet/pods/ec041e3f-8c83-4807-94f2-40a0636a4dd9/volumes"
Oct 29 23:02:20.226237 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:20.226201    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:22.142182 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:22.142145    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:23.615682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:23.615650    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:24.255157 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.255100    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 23:02:24.274239 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.274214    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-socket\") pod \"8d139364-f919-4189-b3fc-cf72ade3e6d6\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") "
Oct 29 23:02:24.274371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.274271    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-data\") pod \"8d139364-f919-4189-b3fc-cf72ade3e6d6\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") "
Oct 29 23:02:24.274371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.274309    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-token\" (UniqueName: \"kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-token\") pod \"8d139364-f919-4189-b3fc-cf72ade3e6d6\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") "
Oct 29 23:02:24.274371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.274351    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-kube-api-access-nzzh9\" (UniqueName: \"kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-kube-api-access-nzzh9\") pod \"8d139364-f919-4189-b3fc-cf72ade3e6d6\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") "
Oct 29 23:02:24.274516 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.274487    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d139364-f919-4189-b3fc-cf72ade3e6d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d139364-f919-4189-b3fc-cf72ade3e6d6-istiod-ca-cert\") pod \"8d139364-f919-4189-b3fc-cf72ade3e6d6\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") "
Oct 29 23:02:24.274582 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.274564    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-podinfo\") pod \"8d139364-f919-4189-b3fc-cf72ade3e6d6\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") "
Oct 29 23:02:24.274683 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.274495    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-socket" pod "8d139364-f919-4189-b3fc-cf72ade3e6d6" (UID: "8d139364-f919-4189-b3fc-cf72ade3e6d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:24.274683 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.274648    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-data" pod "8d139364-f919-4189-b3fc-cf72ade3e6d6" (UID: "8d139364-f919-4189-b3fc-cf72ade3e6d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:24.274865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.274694    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-envoy\") pod \"8d139364-f919-4189-b3fc-cf72ade3e6d6\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") "
Oct 29 23:02:24.274865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.274742    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-certs\") pod \"8d139364-f919-4189-b3fc-cf72ade3e6d6\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") "
Oct 29 23:02:24.274865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.274800    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d139364-f919-4189-b3fc-cf72ade3e6d6-istiod-ca-cert" pod "8d139364-f919-4189-b3fc-cf72ade3e6d6" (UID: "8d139364-f919-4189-b3fc-cf72ade3e6d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:24.274865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.274839    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-credential-socket\") pod \"8d139364-f919-4189-b3fc-cf72ade3e6d6\" (UID: \"8d139364-f919-4189-b3fc-cf72ade3e6d6\") "
Oct 29 23:02:24.275121 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.275076    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-certs" pod "8d139364-f919-4189-b3fc-cf72ade3e6d6" (UID: "8d139364-f919-4189-b3fc-cf72ade3e6d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:24.275265 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.275247    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-credential-socket" pod "8d139364-f919-4189-b3fc-cf72ade3e6d6" (UID: "8d139364-f919-4189-b3fc-cf72ade3e6d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:24.275526 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.275512    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d139364-f919-4189-b3fc-cf72ade3e6d6-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:24.275574 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.275532    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:24.275574 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.275544    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:24.275574 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.275554    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:24.275574 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.275564    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:24.276681 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.276651    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-kube-api-access-nzzh9" pod "8d139364-f919-4189-b3fc-cf72ade3e6d6" (UID: "8d139364-f919-4189-b3fc-cf72ade3e6d6"). InnerVolumeSpecName "kube-api-access-nzzh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:24.277044 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.277016    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-envoy" pod "8d139364-f919-4189-b3fc-cf72ade3e6d6" (UID: "8d139364-f919-4189-b3fc-cf72ade3e6d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:24.277044 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.277022    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-token" pod "8d139364-f919-4189-b3fc-cf72ade3e6d6" (UID: "8d139364-f919-4189-b3fc-cf72ade3e6d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:24.277236 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.277221    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-podinfo" pod "8d139364-f919-4189-b3fc-cf72ade3e6d6" (UID: "8d139364-f919-4189-b3fc-cf72ade3e6d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:24.376931 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.376895    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:24.376931 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.376925    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzzh9\" (UniqueName: \"kubernetes.io/projected/8d139364-f919-4189-b3fc-cf72ade3e6d6-kube-api-access-nzzh9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:24.377116 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.376942    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:24.377116 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.376954    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d139364-f919-4189-b3fc-cf72ade3e6d6-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:24.792852 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.792820    4860 scope.go:122] "RemoveContainer" containerID="e212690949e3a9dd1a7557ccf529de97a05245bb72dcafa9c37666cddb1b621f"
Oct 29 23:02:24.803894 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:24.803872    4860 scope.go:122] "RemoveContainer" containerID="0e763cb8f0b91a4e8cd31d69bb58ee16beb88a5b2281d7121f4029968ddd5e47"
Oct 29 23:02:25.000996 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.000964    4860 generic.go:356] "Generic (PLEG): container finished" podID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerID="61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067" exitCode=0
Oct 29 23:02:25.001113 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.001018    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"
Oct 29 23:02:25.001113 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.001042    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" event={"ID":"8d139364-f919-4189-b3fc-cf72ade3e6d6","Type":"ContainerDied","Data":"61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067"}
Oct 29 23:02:25.001196 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.001112    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm" event={"ID":"8d139364-f919-4189-b3fc-cf72ade3e6d6","Type":"ContainerDied","Data":"dcfe72e95ab809adc3f577060a14da13dc33f20242cdc98b1245c789bd61ceb7"}
Oct 29 23:02:25.001196 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.001135    4860 scope.go:122] "RemoveContainer" containerID="61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067"
Oct 29 23:02:25.015164 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.015146    4860 scope.go:122] "RemoveContainer" containerID="fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6"
Oct 29 23:02:25.025274 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.025252    4860 scope.go:122] "RemoveContainer" containerID="61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067"
Oct 29 23:02:25.025588 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:02:25.025555    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067\": container with ID starting with 61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067 not found: ID does not exist" containerID="61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067"
Oct 29 23:02:25.025646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.025601    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067"} err="failed to get container status \"61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067\": rpc error: code = NotFound desc = could not find container \"61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067\": container with ID starting with 61e682a23a5962d66dd3a92b54aaeabae6f064b1f09f3502638221cc78df7067 not found: ID does not exist"
Oct 29 23:02:25.025646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.025627    4860 scope.go:122] "RemoveContainer" containerID="fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6"
Oct 29 23:02:25.025925 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:02:25.025910    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6\": container with ID starting with fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6 not found: ID does not exist" containerID="fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6"
Oct 29 23:02:25.025969 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.025930    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6"} err="failed to get container status \"fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6\": rpc error: code = NotFound desc = could not find container \"fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6\": container with ID starting with fc1f95343502965951207d162bba3fcd371496eec8595b65e5e01e10aede1fd6 not found: ID does not exist"
Oct 29 23:02:25.072006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.071950    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"]
Oct 29 23:02:25.075579 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.075560    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-63e89c-openshift-default-6d97c8847c-8bqfm"]
Oct 29 23:02:25.338925 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:25.338842    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:26.113060 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:26.113029    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" path="/var/lib/kubelet/pods/8d139364-f919-4189-b3fc-cf72ade3e6d6/volumes"
Oct 29 23:02:26.828838 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:26.828775    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:28.310317 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:28.310287    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:29.504960 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.504924    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"]
Oct 29 23:02:29.505465 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.505180    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj" podUID="00c970b9-cd72-4a45-a8de-317f6d556a5a" containerName="maas-api" containerID="cri-o://bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045" gracePeriod=30
Oct 29 23:02:29.585728 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.585700    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:29.746812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.746673    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:29.807875 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.807843    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 23:02:29.869652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.869620    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:29.943713 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.943686    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-maas-api-tls\") pod \"00c970b9-cd72-4a45-a8de-317f6d556a5a\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") "
Oct 29 23:02:29.943840 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.943757    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-metrics-certs\") pod \"00c970b9-cd72-4a45-a8de-317f6d556a5a\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") "
Oct 29 23:02:29.943909 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.943895    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c970b9-cd72-4a45-a8de-317f6d556a5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c970b9-cd72-4a45-a8de-317f6d556a5a-openshift-service-ca\") pod \"00c970b9-cd72-4a45-a8de-317f6d556a5a\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") "
Oct 29 23:02:29.943958 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.943944    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c970b9-cd72-4a45-a8de-317f6d556a5a-kube-api-access-5zpc4\" (UniqueName: \"kubernetes.io/projected/00c970b9-cd72-4a45-a8de-317f6d556a5a-kube-api-access-5zpc4\") pod \"00c970b9-cd72-4a45-a8de-317f6d556a5a\" (UID: \"00c970b9-cd72-4a45-a8de-317f6d556a5a\") "
Oct 29 23:02:29.944305 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.944279    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c970b9-cd72-4a45-a8de-317f6d556a5a-openshift-service-ca" pod "00c970b9-cd72-4a45-a8de-317f6d556a5a" (UID: "00c970b9-cd72-4a45-a8de-317f6d556a5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:29.944485 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.944468    4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c970b9-cd72-4a45-a8de-317f6d556a5a-openshift-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:29.946022 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.946004    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-metrics-certs" pod "00c970b9-cd72-4a45-a8de-317f6d556a5a" (UID: "00c970b9-cd72-4a45-a8de-317f6d556a5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:29.946345 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.946306    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-maas-api-tls" pod "00c970b9-cd72-4a45-a8de-317f6d556a5a" (UID: "00c970b9-cd72-4a45-a8de-317f6d556a5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:29.946453 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:29.946383    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c970b9-cd72-4a45-a8de-317f6d556a5a-kube-api-access-5zpc4" pod "00c970b9-cd72-4a45-a8de-317f6d556a5a" (UID: "00c970b9-cd72-4a45-a8de-317f6d556a5a"). InnerVolumeSpecName "kube-api-access-5zpc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:30.066401 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.064527    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zpc4\" (UniqueName: \"kubernetes.io/projected/00c970b9-cd72-4a45-a8de-317f6d556a5a-kube-api-access-5zpc4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:30.066401 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.064564    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-maas-api-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:30.066401 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.064577    4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c970b9-cd72-4a45-a8de-317f6d556a5a-metrics-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:30.118375 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.118121    4860 generic.go:356] "Generic (PLEG): container finished" podID="00c970b9-cd72-4a45-a8de-317f6d556a5a" containerID="bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045" exitCode=0
Oct 29 23:02:30.118375 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.118224    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"
Oct 29 23:02:30.122898 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.122860    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj" event={"ID":"00c970b9-cd72-4a45-a8de-317f6d556a5a","Type":"ContainerDied","Data":"bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045"}
Oct 29 23:02:30.123063 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.123044    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj" event={"ID":"00c970b9-cd72-4a45-a8de-317f6d556a5a","Type":"ContainerDied","Data":"dff7193d401735cc2cb1acf02506fea2a58116863999a0646be39c07ab7afb29"}
Oct 29 23:02:30.123192 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.123081    4860 scope.go:122] "RemoveContainer" containerID="bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045"
Oct 29 23:02:30.137378 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.137309    4860 scope.go:122] "RemoveContainer" containerID="bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045"
Oct 29 23:02:30.137697 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:02:30.137670    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045\": container with ID starting with bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045 not found: ID does not exist" containerID="bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045"
Oct 29 23:02:30.137760 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.137709    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045"} err="failed to get container status \"bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045\": rpc error: code = NotFound desc = could not find container \"bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045\": container with ID starting with bf694c34e5156b9664695742f315f8415a7767a1316fbbe3dd9dd3d796dae045 not found: ID does not exist"
Oct 29 23:02:30.174005 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.173975    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"]
Oct 29 23:02:30.177641 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.177619    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-920032-6d9db968bc-8q6bj"]
Oct 29 23:02:30.315697 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:30.315670    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:31.186540 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:31.186508    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:31.316167 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:31.316122    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:31.810449 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:31.810419    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:31.929539 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:31.929505    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:32.113505 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:32.113433    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c970b9-cd72-4a45-a8de-317f6d556a5a" path="/var/lib/kubelet/pods/00c970b9-cd72-4a45-a8de-317f6d556a5a/volumes"
Oct 29 23:02:33.557229 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:33.557198    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:35.023150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:35.023115    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:35.524981 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:35.524947    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:35.642526 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:35.642495    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:35.755774 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:35.755740    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:36.140068 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:36.140036    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:37.256719 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:37.256685    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:37.788504 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:37.788466    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:37.899286 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:37.899253    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:37.925287 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:37.925252    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"]
Oct 29 23:02:37.960475 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:37.960443    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"]
Oct 29 23:02:38.072140 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.072045    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"]
Oct 29 23:02:38.072359 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.072273    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4" podUID="4ad192f3-f04a-439e-8b84-aeb905d0813c" containerName="istio-proxy" containerID="cri-o://81885ba6b8ab1e412b3d3916452c395f7f47abe48d2383c488c996b0aa51bf7e" gracePeriod=30
Oct 29 23:02:38.088747 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.088721    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:38.238152 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.238117    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:38.352354 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.352234    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.615386    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"]
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616339    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616359    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616406    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c6ae3f75-4562-4785-81ab-3c17b887924d" containerName="payload-pre-processing"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616414    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ae3f75-4562-4785-81ab-3c17b887924d" containerName="payload-pre-processing"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616430    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1133119f-4582-4d64-9036-b4ebdae3d993" containerName="maas-api"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616438    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1133119f-4582-4d64-9036-b4ebdae3d993" containerName="maas-api"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616454    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616461    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616478    4860 cpu_manager.go:414] "containerMap: removing container" podUID="00c970b9-cd72-4a45-a8de-317f6d556a5a" containerName="maas-api"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616486    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c970b9-cd72-4a45-a8de-317f6d556a5a" containerName="maas-api"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616498    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4fa5e424-881b-483d-a583-5975412c2e06" containerName="revoke-keys"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616506    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa5e424-881b-483d-a583-5975412c2e06" containerName="revoke-keys"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616534    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616542    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616562    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616569    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616583    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2a4ac286-93cf-4aaf-a3da-80c5f28417bb" containerName="maas-api"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616590    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4ac286-93cf-4aaf-a3da-80c5f28417bb" containerName="maas-api"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616603    4860 cpu_manager.go:414] "containerMap: removing container" podUID="100f773a-93c3-49ef-b704-393f72d7b181" containerName="payload-processing"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616610    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100f773a-93c3-49ef-b704-393f72d7b181" containerName="payload-processing"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616622    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616629    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616646    4860 cpu_manager.go:414] "containerMap: removing container" podUID="5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54" containerName="revoke-keys"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616654    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54" containerName="revoke-keys"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616794    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ae3f75-4562-4785-81ab-3c17b887924d" containerName="payload-pre-processing"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616806    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1133119f-4582-4d64-9036-b4ebdae3d993" containerName="maas-api"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616818    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616828    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c970b9-cd72-4a45-a8de-317f6d556a5a" containerName="maas-api"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616839    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa5e424-881b-483d-a583-5975412c2e06" containerName="revoke-keys"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616851    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616862    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616871    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4ac286-93cf-4aaf-a3da-80c5f28417bb" containerName="maas-api"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616882    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="100f773a-93c3-49ef-b704-393f72d7b181" containerName="payload-processing"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616891    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec041e3f-8c83-4807-94f2-40a0636a4dd9" containerName="istio-proxy"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616902    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa5fef3-7b5a-48c5-abc3-6f21b9e4ec54" containerName="revoke-keys"
Oct 29 23:02:38.617441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.616911    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy"
Oct 29 23:02:38.670019 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.669979    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:38.686935 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.686904    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:02:38.687385 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.687304    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:02:38.695219 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.695183    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"]
Oct 29 23:02:38.787473 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.787430    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5jb8\" (UniqueName: \"kubernetes.io/projected/ba554256-9403-4f32-91d1-212005b78e28-kube-api-access-v5jb8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv\" (UID: \"ba554256-9403-4f32-91d1-212005b78e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:38.787691 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.787600    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba554256-9403-4f32-91d1-212005b78e28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv\" (UID: \"ba554256-9403-4f32-91d1-212005b78e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:38.787739 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.787694    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba554256-9403-4f32-91d1-212005b78e28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv\" (UID: \"ba554256-9403-4f32-91d1-212005b78e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:38.888625 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.888538    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5jb8\" (UniqueName: \"kubernetes.io/projected/ba554256-9403-4f32-91d1-212005b78e28-kube-api-access-v5jb8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv\" (UID: \"ba554256-9403-4f32-91d1-212005b78e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:38.888777 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.888671    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba554256-9403-4f32-91d1-212005b78e28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv\" (UID: \"ba554256-9403-4f32-91d1-212005b78e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:38.888777 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.888748    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba554256-9403-4f32-91d1-212005b78e28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv\" (UID: \"ba554256-9403-4f32-91d1-212005b78e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:38.889606 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.889581    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba554256-9403-4f32-91d1-212005b78e28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv\" (UID: \"ba554256-9403-4f32-91d1-212005b78e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:38.891210 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.891190    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba554256-9403-4f32-91d1-212005b78e28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv\" (UID: \"ba554256-9403-4f32-91d1-212005b78e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:38.908720 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.908697    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5jb8\" (UniqueName: \"kubernetes.io/projected/ba554256-9403-4f32-91d1-212005b78e28-kube-api-access-v5jb8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv\" (UID: \"ba554256-9403-4f32-91d1-212005b78e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:38.989691 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:38.989655    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:39.093805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:39.093719    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:39.293566 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:02:39.293522    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba554256_9403_4f32_91d1_212005b78e28.slice/crio-aa8ead974cf557d63bd5bd4a147fc139d17764be2213be7980a08f44669aa56c WatchSource:0}: Error finding container aa8ead974cf557d63bd5bd4a147fc139d17764be2213be7980a08f44669aa56c: Status 404 returned error can't find the container with id aa8ead974cf557d63bd5bd4a147fc139d17764be2213be7980a08f44669aa56c
Oct 29 23:02:39.295852 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:39.295828    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"]
Oct 29 23:02:40.252298 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:40.252269    4860 generic.go:356] "Generic (PLEG): container finished" podID="ba554256-9403-4f32-91d1-212005b78e28" containerID="32bd5d2d44f131b1a3d910e00c99d161be9d615f485a4eb16c1ffc87fb683784" exitCode=0
Oct 29 23:02:40.252661 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:40.252375    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv" event={"ID":"ba554256-9403-4f32-91d1-212005b78e28","Type":"ContainerDied","Data":"32bd5d2d44f131b1a3d910e00c99d161be9d615f485a4eb16c1ffc87fb683784"}
Oct 29 23:02:40.252661 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:40.252429    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv" event={"ID":"ba554256-9403-4f32-91d1-212005b78e28","Type":"ContainerStarted","Data":"aa8ead974cf557d63bd5bd4a147fc139d17764be2213be7980a08f44669aa56c"}
Oct 29 23:02:40.918445 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:40.918414    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:41.407315 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:41.407290    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:41.525774 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:41.525680    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba554256-9403-4f32-91d1-212005b78e28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba554256-9403-4f32-91d1-212005b78e28-maas-api-openshift-ca\") pod \"ba554256-9403-4f32-91d1-212005b78e28\" (UID: \"ba554256-9403-4f32-91d1-212005b78e28\") "
Oct 29 23:02:41.525957 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:41.525775    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba554256-9403-4f32-91d1-212005b78e28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba554256-9403-4f32-91d1-212005b78e28-maas-api-xks-ca\") pod \"ba554256-9403-4f32-91d1-212005b78e28\" (UID: \"ba554256-9403-4f32-91d1-212005b78e28\") "
Oct 29 23:02:41.526020 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:41.525951    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba554256-9403-4f32-91d1-212005b78e28-kube-api-access-v5jb8\" (UniqueName: \"kubernetes.io/projected/ba554256-9403-4f32-91d1-212005b78e28-kube-api-access-v5jb8\") pod \"ba554256-9403-4f32-91d1-212005b78e28\" (UID: \"ba554256-9403-4f32-91d1-212005b78e28\") "
Oct 29 23:02:41.526106 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:41.526077    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba554256-9403-4f32-91d1-212005b78e28-maas-api-openshift-ca" pod "ba554256-9403-4f32-91d1-212005b78e28" (UID: "ba554256-9403-4f32-91d1-212005b78e28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:41.526562 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:41.526538    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba554256-9403-4f32-91d1-212005b78e28-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:41.527978 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:41.527955    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba554256-9403-4f32-91d1-212005b78e28-maas-api-xks-ca" pod "ba554256-9403-4f32-91d1-212005b78e28" (UID: "ba554256-9403-4f32-91d1-212005b78e28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:41.528124 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:41.528105    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba554256-9403-4f32-91d1-212005b78e28-kube-api-access-v5jb8" pod "ba554256-9403-4f32-91d1-212005b78e28" (UID: "ba554256-9403-4f32-91d1-212005b78e28"). InnerVolumeSpecName "kube-api-access-v5jb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:41.628143 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:41.628113    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba554256-9403-4f32-91d1-212005b78e28-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:41.628143 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:41.628139    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5jb8\" (UniqueName: \"kubernetes.io/projected/ba554256-9403-4f32-91d1-212005b78e28-kube-api-access-v5jb8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:42.185072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.185028    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:42.269357 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.269308    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv" event={"ID":"ba554256-9403-4f32-91d1-212005b78e28","Type":"ContainerDied","Data":"aa8ead974cf557d63bd5bd4a147fc139d17764be2213be7980a08f44669aa56c"}
Oct 29 23:02:42.269547 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.269361    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"
Oct 29 23:02:42.269547 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.269381    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa8ead974cf557d63bd5bd4a147fc139d17764be2213be7980a08f44669aa56c"
Oct 29 23:02:42.326072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.326042    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:42.507258 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.507227    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:42.777130 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.777034    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:42.869862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.869831    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"]
Oct 29 23:02:42.870101 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.870076    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht" podUID="b2e82b87-9570-42a9-b317-27c096817546" containerName="payload-pre-processing" containerID="cri-o://f64436e842025356ccfd9aff65525918eb8eaa203c5473e2dded2801816171ac" gracePeriod=30
Oct 29 23:02:42.925804 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.925735    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn" podUID="e6821c21-38c0-4003-bfe5-66adff17b540" containerName="payload-processing" containerID="cri-o://dd5f9cbf9d0bfc793df3c25f9021ec3bfee05c6953738be9e572c629b01ae06e" gracePeriod=25
Oct 29 23:02:42.948167 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.948124    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:42.961081 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.961047    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77" podUID="2130a25f-43b2-494d-a9f3-e22e9dbfeb73" containerName="payload-pre-processing" containerID="cri-o://0516c48707fbc5022ae6416bc10c6ed978d9ae94dd652e30a9b6dec47b7ccf06" gracePeriod=25
Oct 29 23:02:42.987704 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.987673    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"]
Oct 29 23:02:42.988264 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.988250    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy"
Oct 29 23:02:42.988318 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.988266    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f45ec0f-a12c-4de7-a655-51ba0d0cd85f" containerName="istio-proxy"
Oct 29 23:02:42.988318 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.988287    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ba554256-9403-4f32-91d1-212005b78e28" containerName="revoke-keys"
Oct 29 23:02:42.988318 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.988292    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba554256-9403-4f32-91d1-212005b78e28" containerName="revoke-keys"
Oct 29 23:02:42.988441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.988421    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba554256-9403-4f32-91d1-212005b78e28" containerName="revoke-keys"
Oct 29 23:02:42.988441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.988432    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d139364-f919-4189-b3fc-cf72ade3e6d6" containerName="istio-proxy"
Oct 29 23:02:42.991912 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:42.991896    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:43.011014 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.010990    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"]
Oct 29 23:02:43.023344 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.019628    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"]
Oct 29 23:02:43.029400 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.025069    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:02:43.029400 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.025198    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:02:43.029400 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.027780    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.044265 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.044238    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9d6d4e-openshift-default-dockercfg-km5wx\""
Oct 29 23:02:43.045555 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.045531    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"]
Oct 29 23:02:43.051009 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.050984    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2\" (UID: \"11cbf57f-7841-4033-88d9-80f2853af41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:43.051145 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.051030    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2\" (UID: \"11cbf57f-7841-4033-88d9-80f2853af41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:43.051266 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.051238    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgx5k\" (UniqueName: \"kubernetes.io/projected/11cbf57f-7841-4033-88d9-80f2853af41b-kube-api-access-bgx5k\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2\" (UID: \"11cbf57f-7841-4033-88d9-80f2853af41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:43.076203 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.076173    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:43.153598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.152364    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-data\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.153598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.152439    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgx5k\" (UniqueName: \"kubernetes.io/projected/11cbf57f-7841-4033-88d9-80f2853af41b-kube-api-access-bgx5k\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2\" (UID: \"11cbf57f-7841-4033-88d9-80f2853af41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:43.153598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.152468    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50800f81-4679-4607-accf-3b3920cd73d5-istiod-ca-cert\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.153598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.152589    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50800f81-4679-4607-accf-3b3920cd73d5-istio-podinfo\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.153598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.152622    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-credential-socket\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.153598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.152710    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-envoy\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.153598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.152763    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-certs\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.153598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.152814    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2\" (UID: \"11cbf57f-7841-4033-88d9-80f2853af41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:43.153598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.152856    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2\" (UID: \"11cbf57f-7841-4033-88d9-80f2853af41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:43.153598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.152891    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbgj\" (UniqueName: \"kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-kube-api-access-lkbgj\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.153598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.152931    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-istio-token\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.153598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.153017    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-socket\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.157443 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.155719    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2\" (UID: \"11cbf57f-7841-4033-88d9-80f2853af41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:43.160491 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.160465    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2\" (UID: \"11cbf57f-7841-4033-88d9-80f2853af41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:43.183537 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.183507    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgx5k\" (UniqueName: \"kubernetes.io/projected/11cbf57f-7841-4033-88d9-80f2853af41b-kube-api-access-bgx5k\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2\" (UID: \"11cbf57f-7841-4033-88d9-80f2853af41b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:43.293466 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.291447    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50800f81-4679-4607-accf-3b3920cd73d5-istio-podinfo\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.293466 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.291502    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-credential-socket\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.293466 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.291581    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-envoy\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.293466 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.291623    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-certs\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.293466 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.291686    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbgj\" (UniqueName: \"kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-kube-api-access-lkbgj\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.293466 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.291729    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-istio-token\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.293466 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.291804    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-socket\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.293466 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.291875    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-data\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.293466 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.291925    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50800f81-4679-4607-accf-3b3920cd73d5-istiod-ca-cert\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.293466 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.292949    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50800f81-4679-4607-accf-3b3920cd73d5-istiod-ca-cert\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.303037 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.301196    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-data\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.303037 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.302096    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-certs\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.303037 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.302964    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-socket\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.311661 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.311529    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50800f81-4679-4607-accf-3b3920cd73d5-istio-podinfo\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.311661 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.311567    4860 generic.go:356] "Generic (PLEG): container finished" podID="2130a25f-43b2-494d-a9f3-e22e9dbfeb73" containerID="0516c48707fbc5022ae6416bc10c6ed978d9ae94dd652e30a9b6dec47b7ccf06" exitCode=0
Oct 29 23:02:43.311661 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.311592    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77" event={"ID":"2130a25f-43b2-494d-a9f3-e22e9dbfeb73","Type":"ContainerDied","Data":"0516c48707fbc5022ae6416bc10c6ed978d9ae94dd652e30a9b6dec47b7ccf06"}
Oct 29 23:02:43.331444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.327266    4860 generic.go:356] "Generic (PLEG): container finished" podID="e6821c21-38c0-4003-bfe5-66adff17b540" containerID="dd5f9cbf9d0bfc793df3c25f9021ec3bfee05c6953738be9e572c629b01ae06e" exitCode=0
Oct 29 23:02:43.331444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.328863    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-credential-socket\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.331444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.329059    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn" event={"ID":"e6821c21-38c0-4003-bfe5-66adff17b540","Type":"ContainerDied","Data":"dd5f9cbf9d0bfc793df3c25f9021ec3bfee05c6953738be9e572c629b01ae06e"}
Oct 29 23:02:43.331444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.329297    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-envoy\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.335338 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.333722    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-istio-token\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.339563 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.335998    4860 generic.go:356] "Generic (PLEG): container finished" podID="b2e82b87-9570-42a9-b317-27c096817546" containerID="f64436e842025356ccfd9aff65525918eb8eaa203c5473e2dded2801816171ac" exitCode=0
Oct 29 23:02:43.339563 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.336259    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht" event={"ID":"b2e82b87-9570-42a9-b317-27c096817546","Type":"ContainerDied","Data":"f64436e842025356ccfd9aff65525918eb8eaa203c5473e2dded2801816171ac"}
Oct 29 23:02:43.339563 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.338420    4860 generic.go:356] "Generic (PLEG): container finished" podID="4ad192f3-f04a-439e-8b84-aeb905d0813c" containerID="81885ba6b8ab1e412b3d3916452c395f7f47abe48d2383c488c996b0aa51bf7e" exitCode=0
Oct 29 23:02:43.339563 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.338561    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4" event={"ID":"4ad192f3-f04a-439e-8b84-aeb905d0813c","Type":"ContainerDied","Data":"81885ba6b8ab1e412b3d3916452c395f7f47abe48d2383c488c996b0aa51bf7e"}
Oct 29 23:02:43.340402 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.340353    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbgj\" (UniqueName: \"kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-kube-api-access-lkbgj\") pod \"e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") " pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.364514 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.364435    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"
Oct 29 23:02:43.376361 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.376177    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:43.379625 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.379595    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:43.391780 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.391721    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:43.437769 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.437438    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"
Oct 29 23:02:43.494141 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.494105    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6821c21-38c0-4003-bfe5-66adff17b540-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6821c21-38c0-4003-bfe5-66adff17b540-plugins-config-volume\") pod \"e6821c21-38c0-4003-bfe5-66adff17b540\" (UID: \"e6821c21-38c0-4003-bfe5-66adff17b540\") "
Oct 29 23:02:43.494691 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.494447    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6821c21-38c0-4003-bfe5-66adff17b540-kube-api-access-fz46m\" (UniqueName: \"kubernetes.io/projected/e6821c21-38c0-4003-bfe5-66adff17b540-kube-api-access-fz46m\") pod \"e6821c21-38c0-4003-bfe5-66adff17b540\" (UID: \"e6821c21-38c0-4003-bfe5-66adff17b540\") "
Oct 29 23:02:43.495285 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.495242    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6821c21-38c0-4003-bfe5-66adff17b540-plugins-config-volume" pod "e6821c21-38c0-4003-bfe5-66adff17b540" (UID: "e6821c21-38c0-4003-bfe5-66adff17b540"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:43.503017 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.502991    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:43.504881 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.504617    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"
Oct 29 23:02:43.505451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.505428    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6821c21-38c0-4003-bfe5-66adff17b540-kube-api-access-fz46m" pod "e6821c21-38c0-4003-bfe5-66adff17b540" (UID: "e6821c21-38c0-4003-bfe5-66adff17b540"). InnerVolumeSpecName "kube-api-access-fz46m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:43.586733 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.586706    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 23:02:43.595882 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.595856    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-plugins-config-volume\") pod \"2130a25f-43b2-494d-a9f3-e22e9dbfeb73\" (UID: \"2130a25f-43b2-494d-a9f3-e22e9dbfeb73\") "
Oct 29 23:02:43.596047 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.596029    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e82b87-9570-42a9-b317-27c096817546-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e82b87-9570-42a9-b317-27c096817546-plugins-config-volume\") pod \"b2e82b87-9570-42a9-b317-27c096817546\" (UID: \"b2e82b87-9570-42a9-b317-27c096817546\") "
Oct 29 23:02:43.596202 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.596184    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e82b87-9570-42a9-b317-27c096817546-kube-api-access-sl2r5\" (UniqueName: \"kubernetes.io/projected/b2e82b87-9570-42a9-b317-27c096817546-kube-api-access-sl2r5\") pod \"b2e82b87-9570-42a9-b317-27c096817546\" (UID: \"b2e82b87-9570-42a9-b317-27c096817546\") "
Oct 29 23:02:43.596265 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.596198    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-plugins-config-volume" pod "2130a25f-43b2-494d-a9f3-e22e9dbfeb73" (UID: "2130a25f-43b2-494d-a9f3-e22e9dbfeb73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:43.596374 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.596358    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-kube-api-access-zr8gp\" (UniqueName: \"kubernetes.io/projected/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-kube-api-access-zr8gp\") pod \"2130a25f-43b2-494d-a9f3-e22e9dbfeb73\" (UID: \"2130a25f-43b2-494d-a9f3-e22e9dbfeb73\") "
Oct 29 23:02:43.596446 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.596430    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e82b87-9570-42a9-b317-27c096817546-plugins-config-volume" pod "b2e82b87-9570-42a9-b317-27c096817546" (UID: "b2e82b87-9570-42a9-b317-27c096817546"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:43.596888 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.596868    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.596974 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.596889    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e82b87-9570-42a9-b317-27c096817546-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.596974 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.596898    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz46m\" (UniqueName: \"kubernetes.io/projected/e6821c21-38c0-4003-bfe5-66adff17b540-kube-api-access-fz46m\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.596974 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.596908    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6821c21-38c0-4003-bfe5-66adff17b540-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.598960 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.598935    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e82b87-9570-42a9-b317-27c096817546-kube-api-access-sl2r5" pod "b2e82b87-9570-42a9-b317-27c096817546" (UID: "b2e82b87-9570-42a9-b317-27c096817546"). InnerVolumeSpecName "kube-api-access-sl2r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:43.599723 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.599702    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-kube-api-access-zr8gp" pod "2130a25f-43b2-494d-a9f3-e22e9dbfeb73" (UID: "2130a25f-43b2-494d-a9f3-e22e9dbfeb73"). InnerVolumeSpecName "kube-api-access-zr8gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:43.698205 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698172    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-socket\") pod \"4ad192f3-f04a-439e-8b84-aeb905d0813c\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") "
Oct 29 23:02:43.698434 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698241    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-token\" (UniqueName: \"kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-token\") pod \"4ad192f3-f04a-439e-8b84-aeb905d0813c\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") "
Oct 29 23:02:43.698434 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698275    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-data\") pod \"4ad192f3-f04a-439e-8b84-aeb905d0813c\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") "
Oct 29 23:02:43.698434 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698354    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-certs\") pod \"4ad192f3-f04a-439e-8b84-aeb905d0813c\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") "
Oct 29 23:02:43.698434 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698396    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-envoy\") pod \"4ad192f3-f04a-439e-8b84-aeb905d0813c\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") "
Oct 29 23:02:43.698662 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698459    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-socket" pod "4ad192f3-f04a-439e-8b84-aeb905d0813c" (UID: "4ad192f3-f04a-439e-8b84-aeb905d0813c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.698662 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698484    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-credential-socket\") pod \"4ad192f3-f04a-439e-8b84-aeb905d0813c\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") "
Oct 29 23:02:43.698662 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698538    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-kube-api-access-xtxdh\" (UniqueName: \"kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-kube-api-access-xtxdh\") pod \"4ad192f3-f04a-439e-8b84-aeb905d0813c\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") "
Oct 29 23:02:43.698662 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698575    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-podinfo\") pod \"4ad192f3-f04a-439e-8b84-aeb905d0813c\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") "
Oct 29 23:02:43.698662 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698633    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-certs" pod "4ad192f3-f04a-439e-8b84-aeb905d0813c" (UID: "4ad192f3-f04a-439e-8b84-aeb905d0813c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.698662 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698652    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad192f3-f04a-439e-8b84-aeb905d0813c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad192f3-f04a-439e-8b84-aeb905d0813c-istiod-ca-cert\") pod \"4ad192f3-f04a-439e-8b84-aeb905d0813c\" (UID: \"4ad192f3-f04a-439e-8b84-aeb905d0813c\") "
Oct 29 23:02:43.698975 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698649    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-data" pod "4ad192f3-f04a-439e-8b84-aeb905d0813c" (UID: "4ad192f3-f04a-439e-8b84-aeb905d0813c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.698975 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.698722    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-credential-socket" pod "4ad192f3-f04a-439e-8b84-aeb905d0813c" (UID: "4ad192f3-f04a-439e-8b84-aeb905d0813c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.699086 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.699030    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad192f3-f04a-439e-8b84-aeb905d0813c-istiod-ca-cert" pod "4ad192f3-f04a-439e-8b84-aeb905d0813c" (UID: "4ad192f3-f04a-439e-8b84-aeb905d0813c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:43.699238 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.699202    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.699238 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.699223    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.699238 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.699233    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr8gp\" (UniqueName: \"kubernetes.io/projected/2130a25f-43b2-494d-a9f3-e22e9dbfeb73-kube-api-access-zr8gp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.699478 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.699242    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad192f3-f04a-439e-8b84-aeb905d0813c-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.699478 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.699251    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.699478 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.699258    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl2r5\" (UniqueName: \"kubernetes.io/projected/b2e82b87-9570-42a9-b317-27c096817546-kube-api-access-sl2r5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.699478 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.699266    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.700651 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.700625    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-envoy" pod "4ad192f3-f04a-439e-8b84-aeb905d0813c" (UID: "4ad192f3-f04a-439e-8b84-aeb905d0813c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:43.700867 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.700847    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-token" pod "4ad192f3-f04a-439e-8b84-aeb905d0813c" (UID: "4ad192f3-f04a-439e-8b84-aeb905d0813c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:43.701057 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.701034    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-podinfo" pod "4ad192f3-f04a-439e-8b84-aeb905d0813c" (UID: "4ad192f3-f04a-439e-8b84-aeb905d0813c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:43.701186 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.701064    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-kube-api-access-xtxdh" pod "4ad192f3-f04a-439e-8b84-aeb905d0813c" (UID: "4ad192f3-f04a-439e-8b84-aeb905d0813c"). InnerVolumeSpecName "kube-api-access-xtxdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:43.744788 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:02:43.744766    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cbf57f_7841_4033_88d9_80f2853af41b.slice/crio-b09b80087b7f779f3abbc4633c0e4726b4ffa409d429e6e768635e0dcb982c9a WatchSource:0}: Error finding container b09b80087b7f779f3abbc4633c0e4726b4ffa409d429e6e768635e0dcb982c9a: Status 404 returned error can't find the container with id b09b80087b7f779f3abbc4633c0e4726b4ffa409d429e6e768635e0dcb982c9a
Oct 29 23:02:43.747768 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.747744    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"]
Oct 29 23:02:43.758768 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:02:43.758747    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50800f81_4679_4607_accf_3b3920cd73d5.slice/crio-6d28f6dcbc473a10db560818a58212ad2101aac353cfb9d15d4aa2e86c5ff363 WatchSource:0}: Error finding container 6d28f6dcbc473a10db560818a58212ad2101aac353cfb9d15d4aa2e86c5ff363: Status 404 returned error can't find the container with id 6d28f6dcbc473a10db560818a58212ad2101aac353cfb9d15d4aa2e86c5ff363
Oct 29 23:02:43.761083 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.761053    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:02:43.761184 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.761115    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:02:43.761184 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.761152    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:02:43.763676 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.763656    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"]
Oct 29 23:02:43.800413 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.800393    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.800413 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.800414    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.800529 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.800422    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtxdh\" (UniqueName: \"kubernetes.io/projected/4ad192f3-f04a-439e-8b84-aeb905d0813c-kube-api-access-xtxdh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.800529 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.800431    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad192f3-f04a-439e-8b84-aeb905d0813c-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:43.975798 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:43.975755    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:44.349499 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.348804    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77" event={"ID":"2130a25f-43b2-494d-a9f3-e22e9dbfeb73","Type":"ContainerDied","Data":"500a93f10ea9764f294e79babeacea374ce36f3763aee5532df52fa7b09ec745"}
Oct 29 23:02:44.349499 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.348855    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"
Oct 29 23:02:44.349499 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.348877    4860 scope.go:122] "RemoveContainer" containerID="0516c48707fbc5022ae6416bc10c6ed978d9ae94dd652e30a9b6dec47b7ccf06"
Oct 29 23:02:44.351154 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.351127    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn" event={"ID":"e6821c21-38c0-4003-bfe5-66adff17b540","Type":"ContainerDied","Data":"c8f34fa3f6eaf22562b0f094cddbbd417aa70e390991eb933ec0d47d16a6a3c2"}
Oct 29 23:02:44.351304 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.351242    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"
Oct 29 23:02:44.354033 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.354004    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l" event={"ID":"50800f81-4679-4607-accf-3b3920cd73d5","Type":"ContainerStarted","Data":"a2c20822a3d6792a1b9b02f2a1723337fcae410031a6a989a26670911f2f45d3"}
Oct 29 23:02:44.354152 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.354067    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l" event={"ID":"50800f81-4679-4607-accf-3b3920cd73d5","Type":"ContainerStarted","Data":"6d28f6dcbc473a10db560818a58212ad2101aac353cfb9d15d4aa2e86c5ff363"}
Oct 29 23:02:44.356499 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.356481    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"
Oct 29 23:02:44.356924 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.356904    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht" event={"ID":"b2e82b87-9570-42a9-b317-27c096817546","Type":"ContainerDied","Data":"d42ceaa9ca7864050484bd5794a56879c48cc6062eebc1be93479faa41200533"}
Oct 29 23:02:44.359047 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.359024    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2" event={"ID":"11cbf57f-7841-4033-88d9-80f2853af41b","Type":"ContainerStarted","Data":"9c6fb71d27bdd169ea9d643a39f8fb48f02e401f45b6c4e9b927d2ff96c0bed2"}
Oct 29 23:02:44.359172 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.359080    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2" event={"ID":"11cbf57f-7841-4033-88d9-80f2853af41b","Type":"ContainerStarted","Data":"b09b80087b7f779f3abbc4633c0e4726b4ffa409d429e6e768635e0dcb982c9a"}
Oct 29 23:02:44.360605 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.360587    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"
Oct 29 23:02:44.360605 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.360593    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4" event={"ID":"4ad192f3-f04a-439e-8b84-aeb905d0813c","Type":"ContainerDied","Data":"063372a11f41de02e5fdd7aca8898d24303b7ebf7ab8f5d220f4144e63913f3f"}
Oct 29 23:02:44.368459 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.368441    4860 scope.go:122] "RemoveContainer" containerID="dd5f9cbf9d0bfc793df3c25f9021ec3bfee05c6953738be9e572c629b01ae06e"
Oct 29 23:02:44.379498 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.379475    4860 scope.go:122] "RemoveContainer" containerID="f64436e842025356ccfd9aff65525918eb8eaa203c5473e2dded2801816171ac"
Oct 29 23:02:44.391185 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.391161    4860 scope.go:122] "RemoveContainer" containerID="81885ba6b8ab1e412b3d3916452c395f7f47abe48d2383c488c996b0aa51bf7e"
Oct 29 23:02:44.391992 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.391974    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:44.393986 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.393960    4860 patch_prober.go:29] interesting pod/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 29 23:02:44.394098 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.394013    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l" podUID="50800f81-4679-4607-accf-3b3920cd73d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 29 23:02:44.412589 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.412567    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"]
Oct 29 23:02:44.417272 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.417246    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d6d4e-5c49d979db-z5zzn"]
Oct 29 23:02:44.472430 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.472379    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2" podStartSLOduration=2.472367248 podStartE2EDuration="2.472367248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:44.468901296 +0000 UTC m=+2906.493677448" watchObservedRunningTime="2026-10-29 23:02:44.472367248 +0000 UTC m=+2906.497143380"
Oct 29 23:02:44.507665 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.507637    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"]
Oct 29 23:02:44.513191 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.513167    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85862e-6c4464c9-9xtht"]
Oct 29 23:02:44.559211 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.559116    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l" podStartSLOduration=2.559100338 podStartE2EDuration="2.559100338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:44.554436092 +0000 UTC m=+2906.579212224" watchObservedRunningTime="2026-10-29 23:02:44.559100338 +0000 UTC m=+2906.583876470"
Oct 29 23:02:44.595909 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.595884    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"]
Oct 29 23:02:44.601615 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.601589    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d6d4e-5fd9b5769f-xgn77"]
Oct 29 23:02:44.637426 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.637397    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"]
Oct 29 23:02:44.640807 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:44.640785    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-920032-openshift-default-5b5f9cc67-jzqt4"]
Oct 29 23:02:45.401399 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:45.397138    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:45.401399 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:45.398037    4860 patch_prober.go:29] interesting pod/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 29 23:02:45.401399 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:45.398103    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l" podUID="50800f81-4679-4607-accf-3b3920cd73d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 29 23:02:45.418758 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:45.415498    4860 generic.go:356] "Generic (PLEG): container finished" podID="11cbf57f-7841-4033-88d9-80f2853af41b" containerID="9c6fb71d27bdd169ea9d643a39f8fb48f02e401f45b6c4e9b927d2ff96c0bed2" exitCode=0
Oct 29 23:02:45.418758 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:45.417513    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2" event={"ID":"11cbf57f-7841-4033-88d9-80f2853af41b","Type":"ContainerDied","Data":"9c6fb71d27bdd169ea9d643a39f8fb48f02e401f45b6c4e9b927d2ff96c0bed2"}
Oct 29 23:02:46.112834 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.112798    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2130a25f-43b2-494d-a9f3-e22e9dbfeb73" path="/var/lib/kubelet/pods/2130a25f-43b2-494d-a9f3-e22e9dbfeb73/volumes"
Oct 29 23:02:46.113371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.113356    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad192f3-f04a-439e-8b84-aeb905d0813c" path="/var/lib/kubelet/pods/4ad192f3-f04a-439e-8b84-aeb905d0813c/volumes"
Oct 29 23:02:46.113764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.113749    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e82b87-9570-42a9-b317-27c096817546" path="/var/lib/kubelet/pods/b2e82b87-9570-42a9-b317-27c096817546/volumes"
Oct 29 23:02:46.114437 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.114414    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6821c21-38c0-4003-bfe5-66adff17b540" path="/var/lib/kubelet/pods/e6821c21-38c0-4003-bfe5-66adff17b540/volumes"
Oct 29 23:02:46.392719 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.392629    4860 patch_prober.go:29] interesting pod/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 29 23:02:46.392719 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.392688    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l" podUID="50800f81-4679-4607-accf-3b3920cd73d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 29 23:02:46.635717 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.635689    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:46.747762 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.747728    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-openshift-ca\") pod \"11cbf57f-7841-4033-88d9-80f2853af41b\" (UID: \"11cbf57f-7841-4033-88d9-80f2853af41b\") "
Oct 29 23:02:46.747941 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.747777    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-xks-ca\") pod \"11cbf57f-7841-4033-88d9-80f2853af41b\" (UID: \"11cbf57f-7841-4033-88d9-80f2853af41b\") "
Oct 29 23:02:46.747941 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.747815    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11cbf57f-7841-4033-88d9-80f2853af41b-kube-api-access-bgx5k\" (UniqueName: \"kubernetes.io/projected/11cbf57f-7841-4033-88d9-80f2853af41b-kube-api-access-bgx5k\") pod \"11cbf57f-7841-4033-88d9-80f2853af41b\" (UID: \"11cbf57f-7841-4033-88d9-80f2853af41b\") "
Oct 29 23:02:46.748075 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.748052    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-openshift-ca" pod "11cbf57f-7841-4033-88d9-80f2853af41b" (UID: "11cbf57f-7841-4033-88d9-80f2853af41b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:46.748171 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.748155    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:46.750077 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.750050    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-xks-ca" pod "11cbf57f-7841-4033-88d9-80f2853af41b" (UID: "11cbf57f-7841-4033-88d9-80f2853af41b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:46.750199 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.750135    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cbf57f-7841-4033-88d9-80f2853af41b-kube-api-access-bgx5k" pod "11cbf57f-7841-4033-88d9-80f2853af41b" (UID: "11cbf57f-7841-4033-88d9-80f2853af41b"). InnerVolumeSpecName "kube-api-access-bgx5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:46.849573 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.849543    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11cbf57f-7841-4033-88d9-80f2853af41b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:46.849573 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.849571    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgx5k\" (UniqueName: \"kubernetes.io/projected/11cbf57f-7841-4033-88d9-80f2853af41b-kube-api-access-bgx5k\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:46.909486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:46.909451    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:47.392736 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:47.392698    4860 patch_prober.go:29] interesting pod/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 29 23:02:47.393139 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:47.392761    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l" podUID="50800f81-4679-4607-accf-3b3920cd73d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 29 23:02:47.433409 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:47.433379    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"
Oct 29 23:02:47.433409 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:47.433396    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2" event={"ID":"11cbf57f-7841-4033-88d9-80f2853af41b","Type":"ContainerDied","Data":"b09b80087b7f779f3abbc4633c0e4726b4ffa409d429e6e768635e0dcb982c9a"}
Oct 29 23:02:47.433607 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:47.433442    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09b80087b7f779f3abbc4633c0e4726b4ffa409d429e6e768635e0dcb982c9a"
Oct 29 23:02:48.055367 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:02:48.055333    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:02:48.400045 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:48.397920    4860 patch_prober.go:29] interesting pod/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 29 23:02:48.400045 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:48.397999    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l" podUID="50800f81-4679-4607-accf-3b3920cd73d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 29 23:02:48.488347 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:48.480681    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:49.039011 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:49.038976    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"]
Oct 29 23:02:49.042486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:49.042460    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bedbb1d7-m86b9"]
Oct 29 23:02:49.098483 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:49.097790    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"]
Oct 29 23:02:49.233872 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:49.233829    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:49.348448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:49.345555    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:49.392777 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:49.392743    4860 patch_prober.go:29] interesting pod/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Oct 29 23:02:49.393130 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:49.393082    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l" podUID="50800f81-4679-4607-accf-3b3920cd73d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Oct 29 23:02:50.112615 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:50.112580    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cda08f6-a831-44d0-b9ac-ff84b67331d9" path="/var/lib/kubelet/pods/2cda08f6-a831-44d0-b9ac-ff84b67331d9/volumes"
Oct 29 23:02:50.175825 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:50.175790    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:50.358021 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:50.357984    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:50.433947 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:50.433916    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:50.434593 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:50.434570    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:50.435520 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:50.435498    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:02:50.482079 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:50.482051    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:50.699207 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:50.699124    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:51.177350 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177260    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"]
Oct 29 23:02:51.177815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177799    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e6821c21-38c0-4003-bfe5-66adff17b540" containerName="payload-processing"
Oct 29 23:02:51.177861 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177815    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6821c21-38c0-4003-bfe5-66adff17b540" containerName="payload-processing"
Oct 29 23:02:51.177861 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177830    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4ad192f3-f04a-439e-8b84-aeb905d0813c" containerName="istio-proxy"
Oct 29 23:02:51.177861 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177836    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad192f3-f04a-439e-8b84-aeb905d0813c" containerName="istio-proxy"
Oct 29 23:02:51.177861 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177847    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2130a25f-43b2-494d-a9f3-e22e9dbfeb73" containerName="payload-pre-processing"
Oct 29 23:02:51.177861 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177852    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2130a25f-43b2-494d-a9f3-e22e9dbfeb73" containerName="payload-pre-processing"
Oct 29 23:02:51.178028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177865    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b2e82b87-9570-42a9-b317-27c096817546" containerName="payload-pre-processing"
Oct 29 23:02:51.178028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177870    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e82b87-9570-42a9-b317-27c096817546" containerName="payload-pre-processing"
Oct 29 23:02:51.178028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177894    4860 cpu_manager.go:414] "containerMap: removing container" podUID="11cbf57f-7841-4033-88d9-80f2853af41b" containerName="revoke-keys"
Oct 29 23:02:51.178028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177899    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11cbf57f-7841-4033-88d9-80f2853af41b" containerName="revoke-keys"
Oct 29 23:02:51.178028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177980    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="11cbf57f-7841-4033-88d9-80f2853af41b" containerName="revoke-keys"
Oct 29 23:02:51.178028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.177993    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6821c21-38c0-4003-bfe5-66adff17b540" containerName="payload-processing"
Oct 29 23:02:51.178028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.178001    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad192f3-f04a-439e-8b84-aeb905d0813c" containerName="istio-proxy"
Oct 29 23:02:51.178028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.178007    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2130a25f-43b2-494d-a9f3-e22e9dbfeb73" containerName="payload-pre-processing"
Oct 29 23:02:51.178028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.178013    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e82b87-9570-42a9-b317-27c096817546" containerName="payload-pre-processing"
Oct 29 23:02:51.181722 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.181701    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:51.200586 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.200560    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"]
Oct 29 23:02:51.208354 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.208313    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:02:51.208467 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.208407    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:02:51.330234 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.330203    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h28fk\" (UniqueName: \"kubernetes.io/projected/c889ba3a-96d6-4598-95b0-be4b6654c8d6-kube-api-access-h28fk\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64\" (UID: \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:51.330444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.330312    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64\" (UID: \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:51.330444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.330410    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64\" (UID: \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:51.432260 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.432230    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h28fk\" (UniqueName: \"kubernetes.io/projected/c889ba3a-96d6-4598-95b0-be4b6654c8d6-kube-api-access-h28fk\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64\" (UID: \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:51.432458 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.432310    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64\" (UID: \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:51.432458 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.432354    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64\" (UID: \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:51.433050 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.433026    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64\" (UID: \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:51.434862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.434827    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64\" (UID: \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:51.452362 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.452338    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h28fk\" (UniqueName: \"kubernetes.io/projected/c889ba3a-96d6-4598-95b0-be4b6654c8d6-kube-api-access-h28fk\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64\" (UID: \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:51.495641 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.495614    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:51.794409 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:02:51.794373    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc889ba3a_96d6_4598_95b0_be4b6654c8d6.slice/crio-ddacecf4d4db8dfdaf808a3f7a72629744bb725ec659a3f4519d4008eeba2283 WatchSource:0}: Error finding container ddacecf4d4db8dfdaf808a3f7a72629744bb725ec659a3f4519d4008eeba2283: Status 404 returned error can't find the container with id ddacecf4d4db8dfdaf808a3f7a72629744bb725ec659a3f4519d4008eeba2283
Oct 29 23:02:51.796807 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.796783    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"]
Oct 29 23:02:51.953993 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:51.953958    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:52.475154 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:52.475116    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64" event={"ID":"c889ba3a-96d6-4598-95b0-be4b6654c8d6","Type":"ContainerStarted","Data":"6ea432d0ff3d1e3e7a0eae029bf9b8b0c298b155025c58fc6b5895a04e833bc4"}
Oct 29 23:02:52.475558 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:52.475180    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64" event={"ID":"c889ba3a-96d6-4598-95b0-be4b6654c8d6","Type":"ContainerStarted","Data":"ddacecf4d4db8dfdaf808a3f7a72629744bb725ec659a3f4519d4008eeba2283"}
Oct 29 23:02:52.517152 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:52.517110    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64" podStartSLOduration=1.517095493 podStartE2EDuration="1.517095493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:52.514511454 +0000 UTC m=+2914.539287602" watchObservedRunningTime="2026-10-29 23:02:52.517095493 +0000 UTC m=+2914.541871623"
Oct 29 23:02:53.476093 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:53.476061    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"]
Oct 29 23:02:53.484018 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:53.483991    4860 generic.go:356] "Generic (PLEG): container finished" podID="c889ba3a-96d6-4598-95b0-be4b6654c8d6" containerID="6ea432d0ff3d1e3e7a0eae029bf9b8b0c298b155025c58fc6b5895a04e833bc4" exitCode=0
Oct 29 23:02:53.484163 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:53.484078    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64" event={"ID":"c889ba3a-96d6-4598-95b0-be4b6654c8d6","Type":"ContainerDied","Data":"6ea432d0ff3d1e3e7a0eae029bf9b8b0c298b155025c58fc6b5895a04e833bc4"}
Oct 29 23:02:53.609734 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:53.609704    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:53.728120 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:53.728042    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:54.102546 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:54.102434    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw" podUID="ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032" containerName="payload-pre-processing" containerID="cri-o://ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d" gracePeriod=25
Oct 29 23:02:54.643628 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:54.643602    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:54.787017 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:54.786938    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c889ba3a-96d6-4598-95b0-be4b6654c8d6-kube-api-access-h28fk\" (UniqueName: \"kubernetes.io/projected/c889ba3a-96d6-4598-95b0-be4b6654c8d6-kube-api-access-h28fk\") pod \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\" (UID: \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\") "
Oct 29 23:02:54.787017 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:54.787014    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-openshift-ca\") pod \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\" (UID: \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\") "
Oct 29 23:02:54.787263 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:54.787069    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-xks-ca\") pod \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\" (UID: \"c889ba3a-96d6-4598-95b0-be4b6654c8d6\") "
Oct 29 23:02:54.787414 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:54.787386    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-openshift-ca" pod "c889ba3a-96d6-4598-95b0-be4b6654c8d6" (UID: "c889ba3a-96d6-4598-95b0-be4b6654c8d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:54.789205 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:54.789179    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c889ba3a-96d6-4598-95b0-be4b6654c8d6-kube-api-access-h28fk" pod "c889ba3a-96d6-4598-95b0-be4b6654c8d6" (UID: "c889ba3a-96d6-4598-95b0-be4b6654c8d6"). InnerVolumeSpecName "kube-api-access-h28fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:54.789293 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:54.789251    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-xks-ca" pod "c889ba3a-96d6-4598-95b0-be4b6654c8d6" (UID: "c889ba3a-96d6-4598-95b0-be4b6654c8d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:54.888185 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:54.888158    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h28fk\" (UniqueName: \"kubernetes.io/projected/c889ba3a-96d6-4598-95b0-be4b6654c8d6-kube-api-access-h28fk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:54.888185 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:54.888183    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:54.888320 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:54.888193    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c889ba3a-96d6-4598-95b0-be4b6654c8d6-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.261625 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:55.261597    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:55.500065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:55.500025    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64" event={"ID":"c889ba3a-96d6-4598-95b0-be4b6654c8d6","Type":"ContainerDied","Data":"ddacecf4d4db8dfdaf808a3f7a72629744bb725ec659a3f4519d4008eeba2283"}
Oct 29 23:02:55.500065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:55.500042    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"
Oct 29 23:02:55.500065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:55.500071    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddacecf4d4db8dfdaf808a3f7a72629744bb725ec659a3f4519d4008eeba2283"
Oct 29 23:02:56.445450 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:56.445414    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:56.594262 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:56.594225    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:57.275522 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:57.275491    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:57.875617 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:57.875572    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:57.940782 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:57.940748    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"]
Oct 29 23:02:57.953421 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:57.953392    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02720a-4zhhv"]
Oct 29 23:02:57.995345 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:57.995280    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:58.113319 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:58.113276    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba554256-9403-4f32-91d1-212005b78e28" path="/var/lib/kubelet/pods/ba554256-9403-4f32-91d1-212005b78e28/volumes"
Oct 29 23:02:58.450675 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:58.450642    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:58.476400 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:58.476372    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p" podUID="30d6699e-f439-44ca-879b-4bb1267cd5d3" containerName="payload-pre-processing" containerID="cri-o://49d1b2f0c4025a1cb896a4f5301f5036ef83d21c571e8483d6b1932accf10c43" gracePeriod=25
Oct 29 23:02:58.726539 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:58.726465    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw" podUID="ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:59.035889 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:59.035811    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"]
Oct 29 23:02:59.048364 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:59.048332    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4477c3-g7fkw"]
Oct 29 23:02:59.749769 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:02:59.749683    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:00.113797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:00.113720    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed203352-bea6-4522-a312-a9cf7a03582e" path="/var/lib/kubelet/pods/ed203352-bea6-4522-a312-a9cf7a03582e/volumes"
Oct 29 23:03:01.319852 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:01.319818    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:01.815546 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:01.815510    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:01.952927 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:01.952894    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:02.859373 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:02.859314    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p" podUID="30d6699e-f439-44ca-879b-4bb1267cd5d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:02.974585 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:02.974557    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:04.466385 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:04.466356    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:04.588894 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:04.588851    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:04.735569 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:04.735485    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"]
Oct 29 23:03:04.737423 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:04.735726    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerName="istio-proxy" containerID="cri-o://e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33" gracePeriod=30
Oct 29 23:03:04.794177 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:04.794146    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:05.244960 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.244929    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:05.383390 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.383345    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"]
Oct 29 23:03:05.384090 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.384071    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c889ba3a-96d6-4598-95b0-be4b6654c8d6" containerName="revoke-keys"
Oct 29 23:03:05.384141 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.384093    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c889ba3a-96d6-4598-95b0-be4b6654c8d6" containerName="revoke-keys"
Oct 29 23:03:05.384213 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.384204    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c889ba3a-96d6-4598-95b0-be4b6654c8d6" containerName="revoke-keys"
Oct 29 23:03:05.388985 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.388963    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:05.404458 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.404436    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:03:05.404628 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.404445    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:03:05.404759 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.404566    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"]
Oct 29 23:03:05.555629 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.555550    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2v25\" (UniqueName: \"kubernetes.io/projected/8533ae55-d0b6-4885-9c79-14416574cf0d-kube-api-access-n2v25\") pod \"maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx\" (UID: \"8533ae55-d0b6-4885-9c79-14416574cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:05.555629 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.555586    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx\" (UID: \"8533ae55-d0b6-4885-9c79-14416574cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:05.556075 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.555730    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx\" (UID: \"8533ae55-d0b6-4885-9c79-14416574cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:05.591741 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.591715    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:03:05.591851 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.591759    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:03:05.657204 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.657172    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2v25\" (UniqueName: \"kubernetes.io/projected/8533ae55-d0b6-4885-9c79-14416574cf0d-kube-api-access-n2v25\") pod \"maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx\" (UID: \"8533ae55-d0b6-4885-9c79-14416574cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:05.657350 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.657215    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx\" (UID: \"8533ae55-d0b6-4885-9c79-14416574cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:05.657350 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.657311    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx\" (UID: \"8533ae55-d0b6-4885-9c79-14416574cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:05.658021 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.658000    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx\" (UID: \"8533ae55-d0b6-4885-9c79-14416574cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:05.659709 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.659684    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx\" (UID: \"8533ae55-d0b6-4885-9c79-14416574cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:05.685133 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.685112    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2v25\" (UniqueName: \"kubernetes.io/projected/8533ae55-d0b6-4885-9c79-14416574cf0d-kube-api-access-n2v25\") pod \"maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx\" (UID: \"8533ae55-d0b6-4885-9c79-14416574cf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:05.703796 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:05.703774    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:06.003159 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:03:06.003127    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8533ae55_d0b6_4885_9c79_14416574cf0d.slice/crio-42e2ed4f8c3c3aecd69f073e3dedb7a55b64595b8eac9e942784325bfe3cc134 WatchSource:0}: Error finding container 42e2ed4f8c3c3aecd69f073e3dedb7a55b64595b8eac9e942784325bfe3cc134: Status 404 returned error can't find the container with id 42e2ed4f8c3c3aecd69f073e3dedb7a55b64595b8eac9e942784325bfe3cc134
Oct 29 23:03:06.005504 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:06.005480    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"]
Oct 29 23:03:06.589234 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:06.589146    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx" event={"ID":"8533ae55-d0b6-4885-9c79-14416574cf0d","Type":"ContainerStarted","Data":"fe89a2f66e6ee5ce68676d4e8d03c0f7612d851390e143e7dfda52002f13fe0d"}
Oct 29 23:03:06.589234 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:06.589210    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx" event={"ID":"8533ae55-d0b6-4885-9c79-14416574cf0d","Type":"ContainerStarted","Data":"42e2ed4f8c3c3aecd69f073e3dedb7a55b64595b8eac9e942784325bfe3cc134"}
Oct 29 23:03:06.630000 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:06.629957    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx" podStartSLOduration=1.6299444250000001 podStartE2EDuration="1.629944425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:06.628139877 +0000 UTC m=+2928.652916005" watchObservedRunningTime="2026-10-29 23:03:06.629944425 +0000 UTC m=+2928.654720556"
Oct 29 23:03:06.765785 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:06.765751    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:07.302770 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:07.302740    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:07.601571 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:07.601490    4860 generic.go:356] "Generic (PLEG): container finished" podID="8533ae55-d0b6-4885-9c79-14416574cf0d" containerID="fe89a2f66e6ee5ce68676d4e8d03c0f7612d851390e143e7dfda52002f13fe0d" exitCode=0
Oct 29 23:03:07.601945 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:07.601579    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx" event={"ID":"8533ae55-d0b6-4885-9c79-14416574cf0d","Type":"ContainerDied","Data":"fe89a2f66e6ee5ce68676d4e8d03c0f7612d851390e143e7dfda52002f13fe0d"}
Oct 29 23:03:07.622937 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:07.622901    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:07.634515 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:07.634493    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"]
Oct 29 23:03:07.637935 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:07.637917    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-3ee017-d8gl2"]
Oct 29 23:03:07.736808 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:07.736765    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:08.113803 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:08.113775    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11cbf57f-7841-4033-88d9-80f2853af41b" path="/var/lib/kubelet/pods/11cbf57f-7841-4033-88d9-80f2853af41b/volumes"
Oct 29 23:03:08.119815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:08.119790    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:08.726412 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:08.726376    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw" podUID="ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:08.768691 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:08.768666    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:08.898872 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:08.898800    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8533ae55-d0b6-4885-9c79-14416574cf0d-kube-api-access-n2v25\" (UniqueName: \"kubernetes.io/projected/8533ae55-d0b6-4885-9c79-14416574cf0d-kube-api-access-n2v25\") pod \"8533ae55-d0b6-4885-9c79-14416574cf0d\" (UID: \"8533ae55-d0b6-4885-9c79-14416574cf0d\") "
Oct 29 23:03:08.898872 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:08.898840    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-xks-ca\") pod \"8533ae55-d0b6-4885-9c79-14416574cf0d\" (UID: \"8533ae55-d0b6-4885-9c79-14416574cf0d\") "
Oct 29 23:03:08.899045 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:08.898911    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-openshift-ca\") pod \"8533ae55-d0b6-4885-9c79-14416574cf0d\" (UID: \"8533ae55-d0b6-4885-9c79-14416574cf0d\") "
Oct 29 23:03:08.899353 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:08.899300    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-openshift-ca" pod "8533ae55-d0b6-4885-9c79-14416574cf0d" (UID: "8533ae55-d0b6-4885-9c79-14416574cf0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:08.900956 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:08.900936    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-xks-ca" pod "8533ae55-d0b6-4885-9c79-14416574cf0d" (UID: "8533ae55-d0b6-4885-9c79-14416574cf0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:08.901092 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:08.901074    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8533ae55-d0b6-4885-9c79-14416574cf0d-kube-api-access-n2v25" pod "8533ae55-d0b6-4885-9c79-14416574cf0d" (UID: "8533ae55-d0b6-4885-9c79-14416574cf0d"). InnerVolumeSpecName "kube-api-access-n2v25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:09.000253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:09.000227    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:09.000253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:09.000250    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2v25\" (UniqueName: \"kubernetes.io/projected/8533ae55-d0b6-4885-9c79-14416574cf0d-kube-api-access-n2v25\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:09.000415 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:09.000261    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8533ae55-d0b6-4885-9c79-14416574cf0d-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:09.441733 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:09.441698    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:09.617806 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:09.617779    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"
Oct 29 23:03:09.617987 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:09.617822    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx" event={"ID":"8533ae55-d0b6-4885-9c79-14416574cf0d","Type":"ContainerDied","Data":"42e2ed4f8c3c3aecd69f073e3dedb7a55b64595b8eac9e942784325bfe3cc134"}
Oct 29 23:03:09.617987 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:09.617899    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42e2ed4f8c3c3aecd69f073e3dedb7a55b64595b8eac9e942784325bfe3cc134"
Oct 29 23:03:09.997062 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:09.997035    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 23:03:10.033955 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.033895    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"]
Oct 29 23:03:10.037597 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.037570    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4daa4eb6-8nmqv"]
Oct 29 23:03:10.113687 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.113645    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-token\" (UniqueName: \"kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-token\") pod \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") "
Oct 29 23:03:10.113882 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.113709    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-certs\") pod \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") "
Oct 29 23:03:10.113882 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.113832    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-kube-api-access-7kv6m\" (UniqueName: \"kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-kube-api-access-7kv6m\") pod \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") "
Oct 29 23:03:10.114022 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.113889    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-credential-socket\") pod \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") "
Oct 29 23:03:10.114022 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.113995    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-podinfo\") pod \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") "
Oct 29 23:03:10.114137 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.114024    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-envoy\") pod \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") "
Oct 29 23:03:10.114137 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.114058    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istiod-ca-cert\") pod \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") "
Oct 29 23:03:10.114137 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.114097    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-socket\") pod \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") "
Oct 29 23:03:10.114320 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.114163    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-data\") pod \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\" (UID: \"fe4a8709-3bda-45e4-a052-37b367c4b6c4\") "
Oct 29 23:03:10.114689 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.114532    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3302b1dc-6642-4da6-ab60-b5c9f156f967" path="/var/lib/kubelet/pods/3302b1dc-6642-4da6-ab60-b5c9f156f967/volumes"
Oct 29 23:03:10.114810 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.114746    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-certs" pod "fe4a8709-3bda-45e4-a052-37b367c4b6c4" (UID: "fe4a8709-3bda-45e4-a052-37b367c4b6c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:10.115276 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.115018    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-socket" pod "fe4a8709-3bda-45e4-a052-37b367c4b6c4" (UID: "fe4a8709-3bda-45e4-a052-37b367c4b6c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:10.115276 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.115079    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-credential-socket" pod "fe4a8709-3bda-45e4-a052-37b367c4b6c4" (UID: "fe4a8709-3bda-45e4-a052-37b367c4b6c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:10.115276 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.115236    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istiod-ca-cert" pod "fe4a8709-3bda-45e4-a052-37b367c4b6c4" (UID: "fe4a8709-3bda-45e4-a052-37b367c4b6c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:10.115638 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.115613    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-data" pod "fe4a8709-3bda-45e4-a052-37b367c4b6c4" (UID: "fe4a8709-3bda-45e4-a052-37b367c4b6c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:10.116253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.116232    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.117185 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.117167    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.117354 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.117339    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.117467 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.117453    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.117575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.117563    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.120850 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.120816    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-kube-api-access-7kv6m" pod "fe4a8709-3bda-45e4-a052-37b367c4b6c4" (UID: "fe4a8709-3bda-45e4-a052-37b367c4b6c4"). InnerVolumeSpecName "kube-api-access-7kv6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:10.121120 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.121095    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-podinfo" pod "fe4a8709-3bda-45e4-a052-37b367c4b6c4" (UID: "fe4a8709-3bda-45e4-a052-37b367c4b6c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:10.121449 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.121414    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-envoy" pod "fe4a8709-3bda-45e4-a052-37b367c4b6c4" (UID: "fe4a8709-3bda-45e4-a052-37b367c4b6c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:10.125500 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.125196    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-token" pod "fe4a8709-3bda-45e4-a052-37b367c4b6c4" (UID: "fe4a8709-3bda-45e4-a052-37b367c4b6c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:10.218642 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.218615    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.218642 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.218641    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kv6m\" (UniqueName: \"kubernetes.io/projected/fe4a8709-3bda-45e4-a052-37b367c4b6c4-kube-api-access-7kv6m\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.218892 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.218652    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.218892 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.218662    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4a8709-3bda-45e4-a052-37b367c4b6c4-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:10.309794 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.309723    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:10.627725 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.627648    4860 generic.go:356] "Generic (PLEG): container finished" podID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerID="e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33" exitCode=0
Oct 29 23:03:10.627725 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.627700    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"
Oct 29 23:03:10.627941 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.627737    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" event={"ID":"fe4a8709-3bda-45e4-a052-37b367c4b6c4","Type":"ContainerDied","Data":"e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33"}
Oct 29 23:03:10.627941 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.627794    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w" event={"ID":"fe4a8709-3bda-45e4-a052-37b367c4b6c4","Type":"ContainerDied","Data":"dc21b4b3cb5f677c5f3aa22776bf46151e2af97cdc8e2c5e8664226f9a073a44"}
Oct 29 23:03:10.627941 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.627825    4860 scope.go:122] "RemoveContainer" containerID="e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33"
Oct 29 23:03:10.639133 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.639114    4860 scope.go:122] "RemoveContainer" containerID="e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33"
Oct 29 23:03:10.639459 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:10.639439    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33\": container with ID starting with e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33 not found: ID does not exist" containerID="e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33"
Oct 29 23:03:10.639521 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.639467    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33"} err="failed to get container status \"e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33\": rpc error: code = NotFound desc = could not find container \"e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33\": container with ID starting with e526cb5dfb7434ebedec21110651df3118bf67cca653060a1bb6945fee9c5d33 not found: ID does not exist"
Oct 29 23:03:10.682727 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.682703    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"]
Oct 29 23:03:10.695898 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.695873    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-02720a-openshift-default-6b64cb69-sgz2w"]
Oct 29 23:03:10.792694 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:10.792673    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:11.027091 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:11.027066    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:12.113744 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:12.113714    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" path="/var/lib/kubelet/pods/fe4a8709-3bda-45e4-a052-37b367c4b6c4/volumes"
Oct 29 23:03:12.625243 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:12.625209    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:12.859798 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:12.859761    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p" podUID="30d6699e-f439-44ca-879b-4bb1267cd5d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:13.901175 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:13.901144    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:14.054384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.054355    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:14.303487 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.303433    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:14.359811 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.359779    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"]
Oct 29 23:03:14.360058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.360032    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy" containerID="cri-o://1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301" gracePeriod=30
Oct 29 23:03:14.433046 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.433011    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:14.474463 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.474437    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"
Oct 29 23:03:14.526808 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.526729    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"]
Oct 29 23:03:14.529957 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.529930    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-9aafe6-zlj64"]
Oct 29 23:03:14.549974 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.549947    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:14.570010 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.569989    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-kube-api-access-m8wfx\" (UniqueName: \"kubernetes.io/projected/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-kube-api-access-m8wfx\") pod \"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032\" (UID: \"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032\") "
Oct 29 23:03:14.570242 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.570223    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-plugins-config-volume\") pod \"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032\" (UID: \"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032\") "
Oct 29 23:03:14.570605 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.570580    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-plugins-config-volume" pod "ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032" (UID: "ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:14.570741 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.570725    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.572290 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.572265    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-kube-api-access-m8wfx" pod "ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032" (UID: "ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032"). InnerVolumeSpecName "kube-api-access-m8wfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:14.672109 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.671876    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8wfx\" (UniqueName: \"kubernetes.io/projected/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032-kube-api-access-m8wfx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:14.676960 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.676878    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:14.686192 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.686161    4860 generic.go:356] "Generic (PLEG): container finished" podID="ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032" containerID="ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d" exitCode=0
Oct 29 23:03:14.686389 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.686217    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"
Oct 29 23:03:14.686389 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.686248    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw" event={"ID":"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032","Type":"ContainerDied","Data":"ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d"}
Oct 29 23:03:14.686389 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.686306    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw" event={"ID":"ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032","Type":"ContainerDied","Data":"f971dc3f505ace94b03287721ba1c16066360d7b7694c88d7900e8e2b3221e38"}
Oct 29 23:03:14.686389 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.686343    4860 scope.go:122] "RemoveContainer" containerID="ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d"
Oct 29 23:03:14.703164 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.703143    4860 scope.go:122] "RemoveContainer" containerID="ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d"
Oct 29 23:03:14.703468 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:14.703447    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d\": container with ID starting with ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d not found: ID does not exist" containerID="ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d"
Oct 29 23:03:14.703517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.703477    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d"} err="failed to get container status \"ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d\": rpc error: code = NotFound desc = could not find container \"ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d\": container with ID starting with ae68acb2965adce5843281d20a0c8d488969e18cd55247fb763e5126b2e8360d not found: ID does not exist"
Oct 29 23:03:14.743519 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.743497    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"]
Oct 29 23:03:14.749171 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.749151    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02720a-54d9b599c9-q5xcw"]
Oct 29 23:03:14.896505 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:14.896281    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:15.052317 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:15.052277    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:15.336879 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:15.336838    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:15.869196 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:15.869163    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"]
Oct 29 23:03:16.038143 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:16.038083    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:16.140871 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:16.140784    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032" path="/var/lib/kubelet/pods/ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032/volumes"
Oct 29 23:03:16.141308 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:16.141285    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c889ba3a-96d6-4598-95b0-be4b6654c8d6" path="/var/lib/kubelet/pods/c889ba3a-96d6-4598-95b0-be4b6654c8d6/volumes"
Oct 29 23:03:16.158268 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:16.158241    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:17.024565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:17.024534    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:17.145900 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:17.145865    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:17.537542 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:17.537517    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-8wn8h_23a26fe4-1010-464f-9df0-ff85860cc488/payload-processing/0.log"
Oct 29 23:03:17.786501 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:17.786475    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-8wn8h_23a26fe4-1010-464f-9df0-ff85860cc488/payload-processing/0.log"
Oct 29 23:03:17.819799 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:17.819736    4860 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:03:17.819902 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:17.819794    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:03:18.060848 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:18.060814    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:03:18.577430 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.577400    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"]
Oct 29 23:03:18.577994 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.577978    4860 cpu_manager.go:414] "containerMap: removing container" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerName="istio-proxy"
Oct 29 23:03:18.578051 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.577995    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerName="istio-proxy"
Oct 29 23:03:18.578051 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.578016    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032" containerName="payload-pre-processing"
Oct 29 23:03:18.578051 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.578021    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032" containerName="payload-pre-processing"
Oct 29 23:03:18.578051 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.578036    4860 cpu_manager.go:414] "containerMap: removing container" podUID="8533ae55-d0b6-4885-9c79-14416574cf0d" containerName="revoke-keys"
Oct 29 23:03:18.578051 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.578041    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8533ae55-d0b6-4885-9c79-14416574cf0d" containerName="revoke-keys"
Oct 29 23:03:18.578253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.578133    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe4a8709-3bda-45e4-a052-37b367c4b6c4" containerName="istio-proxy"
Oct 29 23:03:18.578253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.578146    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5f9159-7bf8-48e0-8fe5-b3cdd3a5a032" containerName="payload-pre-processing"
Oct 29 23:03:18.578253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.578154    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="8533ae55-d0b6-4885-9c79-14416574cf0d" containerName="revoke-keys"
Oct 29 23:03:18.581557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.581533    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:18.600691 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.600666    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"]
Oct 29 23:03:18.608443 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.608419    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:03:18.608561 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.608483    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:03:18.724253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.724220    4860 generic.go:356] "Generic (PLEG): container finished" podID="30d6699e-f439-44ca-879b-4bb1267cd5d3" containerID="49d1b2f0c4025a1cb896a4f5301f5036ef83d21c571e8483d6b1932accf10c43" exitCode=0
Oct 29 23:03:18.724473 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.724457    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p" event={"ID":"30d6699e-f439-44ca-879b-4bb1267cd5d3","Type":"ContainerDied","Data":"49d1b2f0c4025a1cb896a4f5301f5036ef83d21c571e8483d6b1932accf10c43"}
Oct 29 23:03:18.724610 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.724586    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w\" (UID: \"6da5e98e-cace-47e5-85b9-49142092cb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:18.724786 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.724761    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6tv5\" (UniqueName: \"kubernetes.io/projected/6da5e98e-cace-47e5-85b9-49142092cb0b-kube-api-access-z6tv5\") pod \"maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w\" (UID: \"6da5e98e-cace-47e5-85b9-49142092cb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:18.725032 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.725009    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w\" (UID: \"6da5e98e-cace-47e5-85b9-49142092cb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:18.735374 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.735272    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:18.793028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.792994    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"
Oct 29 23:03:18.826699 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.826668    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6tv5\" (UniqueName: \"kubernetes.io/projected/6da5e98e-cace-47e5-85b9-49142092cb0b-kube-api-access-z6tv5\") pod \"maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w\" (UID: \"6da5e98e-cace-47e5-85b9-49142092cb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:18.826892 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.826782    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w\" (UID: \"6da5e98e-cace-47e5-85b9-49142092cb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:18.827001 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.826901    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w\" (UID: \"6da5e98e-cace-47e5-85b9-49142092cb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:18.828028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.827999    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w\" (UID: \"6da5e98e-cace-47e5-85b9-49142092cb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:18.829934 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.829912    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w\" (UID: \"6da5e98e-cace-47e5-85b9-49142092cb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:18.847440 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.847410    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6tv5\" (UniqueName: \"kubernetes.io/projected/6da5e98e-cace-47e5-85b9-49142092cb0b-kube-api-access-z6tv5\") pod \"maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w\" (UID: \"6da5e98e-cace-47e5-85b9-49142092cb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:18.896721 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.896695    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:18.927660 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.927637    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d6699e-f439-44ca-879b-4bb1267cd5d3-kube-api-access-vzjmt\" (UniqueName: \"kubernetes.io/projected/30d6699e-f439-44ca-879b-4bb1267cd5d3-kube-api-access-vzjmt\") pod \"30d6699e-f439-44ca-879b-4bb1267cd5d3\" (UID: \"30d6699e-f439-44ca-879b-4bb1267cd5d3\") "
Oct 29 23:03:18.927834 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.927818    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d6699e-f439-44ca-879b-4bb1267cd5d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d6699e-f439-44ca-879b-4bb1267cd5d3-plugins-config-volume\") pod \"30d6699e-f439-44ca-879b-4bb1267cd5d3\" (UID: \"30d6699e-f439-44ca-879b-4bb1267cd5d3\") "
Oct 29 23:03:18.928185 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.928159    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d6699e-f439-44ca-879b-4bb1267cd5d3-plugins-config-volume" pod "30d6699e-f439-44ca-879b-4bb1267cd5d3" (UID: "30d6699e-f439-44ca-879b-4bb1267cd5d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:18.929717 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:18.929692    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d6699e-f439-44ca-879b-4bb1267cd5d3-kube-api-access-vzjmt" pod "30d6699e-f439-44ca-879b-4bb1267cd5d3" (UID: "30d6699e-f439-44ca-879b-4bb1267cd5d3"). InnerVolumeSpecName "kube-api-access-vzjmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:19.029923 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.029851    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzjmt\" (UniqueName: \"kubernetes.io/projected/30d6699e-f439-44ca-879b-4bb1267cd5d3-kube-api-access-vzjmt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.029923 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.029878    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d6699e-f439-44ca-879b-4bb1267cd5d3-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.198438 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:03:19.198402    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da5e98e_cace_47e5_85b9_49142092cb0b.slice/crio-eac0079e540279d97085970c8cd261fdea6045e85e9e15e45deb2a76f314159e WatchSource:0}: Error finding container eac0079e540279d97085970c8cd261fdea6045e85e9e15e45deb2a76f314159e: Status 404 returned error can't find the container with id eac0079e540279d97085970c8cd261fdea6045e85e9e15e45deb2a76f314159e
Oct 29 23:03:19.201454 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.201426    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"]
Oct 29 23:03:19.641362 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.641293    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 23:03:19.736013 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.735989    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"
Oct 29 23:03:19.736192 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.736032    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p" event={"ID":"30d6699e-f439-44ca-879b-4bb1267cd5d3","Type":"ContainerDied","Data":"ea7d1d1342bec06a5fbb0fc2987c54e0ea1959d96edcc02fb597289e7624c48d"}
Oct 29 23:03:19.736192 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.736093    4860 scope.go:122] "RemoveContainer" containerID="49d1b2f0c4025a1cb896a4f5301f5036ef83d21c571e8483d6b1932accf10c43"
Oct 29 23:03:19.736480 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.736441    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-envoy\") pod \"33379dd5-161b-4506-9860-228572c35c00\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") "
Oct 29 23:03:19.736616 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.736496    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-certs\") pod \"33379dd5-161b-4506-9860-228572c35c00\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") "
Oct 29 23:03:19.736616 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.736540    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-istio-token\" (UniqueName: \"kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-istio-token\") pod \"33379dd5-161b-4506-9860-228572c35c00\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") "
Oct 29 23:03:19.736758 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.736628    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-credential-socket\") pod \"33379dd5-161b-4506-9860-228572c35c00\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") "
Oct 29 23:03:19.736829 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.736786    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-certs" pod "33379dd5-161b-4506-9860-228572c35c00" (UID: "33379dd5-161b-4506-9860-228572c35c00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.737043 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.737002    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-credential-socket" pod "33379dd5-161b-4506-9860-228572c35c00" (UID: "33379dd5-161b-4506-9860-228572c35c00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.737165 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.737037    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-kube-api-access-jndc9\" (UniqueName: \"kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-kube-api-access-jndc9\") pod \"33379dd5-161b-4506-9860-228572c35c00\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") "
Oct 29 23:03:19.737165 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.737083    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33379dd5-161b-4506-9860-228572c35c00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33379dd5-161b-4506-9860-228572c35c00-istiod-ca-cert\") pod \"33379dd5-161b-4506-9860-228572c35c00\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") "
Oct 29 23:03:19.737165 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.737128    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33379dd5-161b-4506-9860-228572c35c00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33379dd5-161b-4506-9860-228572c35c00-istio-podinfo\") pod \"33379dd5-161b-4506-9860-228572c35c00\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") "
Oct 29 23:03:19.737357 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.737336    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-socket\") pod \"33379dd5-161b-4506-9860-228572c35c00\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") "
Oct 29 23:03:19.737423 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.737377    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-data\") pod \"33379dd5-161b-4506-9860-228572c35c00\" (UID: \"33379dd5-161b-4506-9860-228572c35c00\") "
Oct 29 23:03:19.737793 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.737723    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33379dd5-161b-4506-9860-228572c35c00-istiod-ca-cert" pod "33379dd5-161b-4506-9860-228572c35c00" (UID: "33379dd5-161b-4506-9860-228572c35c00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:19.737927 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.737807    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.737927 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.737829    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.737927 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.737899    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-data" pod "33379dd5-161b-4506-9860-228572c35c00" (UID: "33379dd5-161b-4506-9860-228572c35c00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.738119 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.738071    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-socket" pod "33379dd5-161b-4506-9860-228572c35c00" (UID: "33379dd5-161b-4506-9860-228572c35c00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.739236 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.739208    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-envoy" pod "33379dd5-161b-4506-9860-228572c35c00" (UID: "33379dd5-161b-4506-9860-228572c35c00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:19.739569 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.739545    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-istio-token" pod "33379dd5-161b-4506-9860-228572c35c00" (UID: "33379dd5-161b-4506-9860-228572c35c00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:19.740196 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.740172    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-kube-api-access-jndc9" pod "33379dd5-161b-4506-9860-228572c35c00" (UID: "33379dd5-161b-4506-9860-228572c35c00"). InnerVolumeSpecName "kube-api-access-jndc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:19.740564 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.740519    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33379dd5-161b-4506-9860-228572c35c00-istio-podinfo" pod "33379dd5-161b-4506-9860-228572c35c00" (UID: "33379dd5-161b-4506-9860-228572c35c00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:19.740886 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.740781    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w" event={"ID":"6da5e98e-cace-47e5-85b9-49142092cb0b","Type":"ContainerStarted","Data":"a7a6098d0223ba5f0e468c6605a6dee7dedc5964ed426cbe2500ca18c4bb0692"}
Oct 29 23:03:19.740886 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.740846    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w" event={"ID":"6da5e98e-cace-47e5-85b9-49142092cb0b","Type":"ContainerStarted","Data":"eac0079e540279d97085970c8cd261fdea6045e85e9e15e45deb2a76f314159e"}
Oct 29 23:03:19.743462 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.743435    4860 generic.go:356] "Generic (PLEG): container finished" podID="33379dd5-161b-4506-9860-228572c35c00" containerID="1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301" exitCode=0
Oct 29 23:03:19.743546 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.743516    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" event={"ID":"33379dd5-161b-4506-9860-228572c35c00","Type":"ContainerDied","Data":"1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301"}
Oct 29 23:03:19.743613 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.743561    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw" event={"ID":"33379dd5-161b-4506-9860-228572c35c00","Type":"ContainerDied","Data":"06e09797c5d1691e3b58d010e7b63c841c4424e16d421666b35bcc9bfeacef88"}
Oct 29 23:03:19.743674 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.743634    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"
Oct 29 23:03:19.769868 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.769851    4860 scope.go:122] "RemoveContainer" containerID="1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301"
Oct 29 23:03:19.779677 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.779661    4860 scope.go:122] "RemoveContainer" containerID="5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013"
Oct 29 23:03:19.789634 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.789615    4860 scope.go:122] "RemoveContainer" containerID="1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301"
Oct 29 23:03:19.789939 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:19.789920    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301\": container with ID starting with 1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301 not found: ID does not exist" containerID="1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301"
Oct 29 23:03:19.790005 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.789947    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301"} err="failed to get container status \"1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301\": rpc error: code = NotFound desc = could not find container \"1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301\": container with ID starting with 1474b40ff82a19d458425447d9eccce8d94278ee1c7f8f4c105f1f703f795301 not found: ID does not exist"
Oct 29 23:03:19.790005 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.789965    4860 scope.go:122] "RemoveContainer" containerID="5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013"
Oct 29 23:03:19.790218 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:19.790200    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013\": container with ID starting with 5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013 not found: ID does not exist" containerID="5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013"
Oct 29 23:03:19.790274 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.790224    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013"} err="failed to get container status \"5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013\": rpc error: code = NotFound desc = could not find container \"5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013\": container with ID starting with 5439bf4722f90ec3055225e5efe24673367bab051d7d3f25f55d121090220013 not found: ID does not exist"
Oct 29 23:03:19.794881 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.794836    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w" podStartSLOduration=1.794825296 podStartE2EDuration="1.794825296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:19.79202746 +0000 UTC m=+2941.816803591" watchObservedRunningTime="2026-10-29 23:03:19.794825296 +0000 UTC m=+2941.819601427"
Oct 29 23:03:19.834730 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.834706    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"]
Oct 29 23:03:19.839243 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.839217    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.839341 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.839248    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.839341 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.839262    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33379dd5-161b-4506-9860-228572c35c00-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.839341 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.839275    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.839341 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.839289    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jndc9\" (UniqueName: \"kubernetes.io/projected/33379dd5-161b-4506-9860-228572c35c00-kube-api-access-jndc9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.839341 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.839303    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33379dd5-161b-4506-9860-228572c35c00-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.839341 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.839316    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33379dd5-161b-4506-9860-228572c35c00-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.844805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.844786    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"]
Oct 29 23:03:19.856299 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.856276    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"]
Oct 29 23:03:19.861062 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.861041    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3ee017-openshift-default-86b45fc4d7-hjvtw"]
Oct 29 23:03:19.902790 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.902687    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"]
Oct 29 23:03:19.907013 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.906994    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3ee017-b5cc8c7ff-nnl9p"]
Oct 29 23:03:19.960664 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:19.960634    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:20.113395 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:20.113366    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d6699e-f439-44ca-879b-4bb1267cd5d3" path="/var/lib/kubelet/pods/30d6699e-f439-44ca-879b-4bb1267cd5d3/volumes"
Oct 29 23:03:20.113876 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:20.113863    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33379dd5-161b-4506-9860-228572c35c00" path="/var/lib/kubelet/pods/33379dd5-161b-4506-9860-228572c35c00/volumes"
Oct 29 23:03:20.445676 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:20.445647    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:20.756197 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:20.756107    4860 generic.go:356] "Generic (PLEG): container finished" podID="6da5e98e-cace-47e5-85b9-49142092cb0b" containerID="a7a6098d0223ba5f0e468c6605a6dee7dedc5964ed426cbe2500ca18c4bb0692" exitCode=0
Oct 29 23:03:20.756197 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:20.756186    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w" event={"ID":"6da5e98e-cace-47e5-85b9-49142092cb0b","Type":"ContainerDied","Data":"a7a6098d0223ba5f0e468c6605a6dee7dedc5964ed426cbe2500ca18c4bb0692"}
Oct 29 23:03:20.869878 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:20.869843    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg" podUID="646e1a1e-8f19-4df7-91d8-4864270cae7d" containerName="payload-processing" containerID="cri-o://f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5" gracePeriod=25
Oct 29 23:03:21.032845 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:21.032764    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"]
Oct 29 23:03:21.036127 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:21.036096    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9eba808e-qj8tm"]
Oct 29 23:03:21.431458 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:21.431421    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:21.672143 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:21.672094    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:21.775685 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:21.775633    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"]
Oct 29 23:03:21.776539 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:21.776250    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy" containerID="cri-o://7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646" gracePeriod=30
Oct 29 23:03:21.818709 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:21.818656    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:21.973852 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:21.973827    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:22.096542 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.096469    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-openshift-ca\") pod \"6da5e98e-cace-47e5-85b9-49142092cb0b\" (UID: \"6da5e98e-cace-47e5-85b9-49142092cb0b\") "
Oct 29 23:03:22.096824 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.096560    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da5e98e-cace-47e5-85b9-49142092cb0b-kube-api-access-z6tv5\" (UniqueName: \"kubernetes.io/projected/6da5e98e-cace-47e5-85b9-49142092cb0b-kube-api-access-z6tv5\") pod \"6da5e98e-cace-47e5-85b9-49142092cb0b\" (UID: \"6da5e98e-cace-47e5-85b9-49142092cb0b\") "
Oct 29 23:03:22.096824 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.096688    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-xks-ca\") pod \"6da5e98e-cace-47e5-85b9-49142092cb0b\" (UID: \"6da5e98e-cace-47e5-85b9-49142092cb0b\") "
Oct 29 23:03:22.096956 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.096852    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-openshift-ca" pod "6da5e98e-cace-47e5-85b9-49142092cb0b" (UID: "6da5e98e-cace-47e5-85b9-49142092cb0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:22.097202 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.097180    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.098914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.098890    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-xks-ca" pod "6da5e98e-cace-47e5-85b9-49142092cb0b" (UID: "6da5e98e-cace-47e5-85b9-49142092cb0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:22.098914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.098899    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da5e98e-cace-47e5-85b9-49142092cb0b-kube-api-access-z6tv5" pod "6da5e98e-cace-47e5-85b9-49142092cb0b" (UID: "6da5e98e-cace-47e5-85b9-49142092cb0b"). InnerVolumeSpecName "kube-api-access-z6tv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:22.113107 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.113083    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b27c2001-f753-4594-835a-c75f45db3ed1" path="/var/lib/kubelet/pods/b27c2001-f753-4594-835a-c75f45db3ed1/volumes"
Oct 29 23:03:22.198092 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.198064    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da5e98e-cace-47e5-85b9-49142092cb0b-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.198092 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.198090    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6tv5\" (UniqueName: \"kubernetes.io/projected/6da5e98e-cace-47e5-85b9-49142092cb0b-kube-api-access-z6tv5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:22.386714 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.386637    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:22.775670 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.775641    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"
Oct 29 23:03:22.775838 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.775663    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w" event={"ID":"6da5e98e-cace-47e5-85b9-49142092cb0b","Type":"ContainerDied","Data":"eac0079e540279d97085970c8cd261fdea6045e85e9e15e45deb2a76f314159e"}
Oct 29 23:03:22.775838 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:22.775712    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eac0079e540279d97085970c8cd261fdea6045e85e9e15e45deb2a76f314159e"
Oct 29 23:03:24.833145 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:24.833110    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:24.888034 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:24.888005    4860 scope.go:122] "RemoveContainer" containerID="669c72661c6ea46b39c86c74db4bac9b0dada230230e8b5438684c89f76a5aef"
Oct 29 23:03:24.898351 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:24.898308    4860 scope.go:122] "RemoveContainer" containerID="605bde6ee38f50beff6ee1fb2b34a59fed049474101b35416f33be12652b195f"
Oct 29 23:03:24.907589 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:24.907568    4860 scope.go:122] "RemoveContainer" containerID="23a252cbc4648c4278e756ac30f478c2c0fd58720c8239e56b02f3837fc1fbe4"
Oct 29 23:03:25.507007 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:25.506973    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"]
Oct 29 23:03:25.683224 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:25.683182    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:25.803009 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:25.802916    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:26.534625 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:26.534591    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:27.066882 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.066819    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 23:03:27.195231 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195198    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8790a27-ef61-43db-803b-fc295959b33a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8790a27-ef61-43db-803b-fc295959b33a-istiod-ca-cert\") pod \"c8790a27-ef61-43db-803b-fc295959b33a\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") "
Oct 29 23:03:27.195428 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195246    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-istio-token\" (UniqueName: \"kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-istio-token\") pod \"c8790a27-ef61-43db-803b-fc295959b33a\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") "
Oct 29 23:03:27.195428 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195306    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-credential-socket\") pod \"c8790a27-ef61-43db-803b-fc295959b33a\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") "
Oct 29 23:03:27.195428 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195413    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8790a27-ef61-43db-803b-fc295959b33a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8790a27-ef61-43db-803b-fc295959b33a-istio-podinfo\") pod \"c8790a27-ef61-43db-803b-fc295959b33a\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") "
Oct 29 23:03:27.195634 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195447    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-envoy\") pod \"c8790a27-ef61-43db-803b-fc295959b33a\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") "
Oct 29 23:03:27.195634 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195539    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-kube-api-access-q6vlm\" (UniqueName: \"kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-kube-api-access-q6vlm\") pod \"c8790a27-ef61-43db-803b-fc295959b33a\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") "
Oct 29 23:03:27.195634 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195583    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-certs\") pod \"c8790a27-ef61-43db-803b-fc295959b33a\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") "
Oct 29 23:03:27.195634 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195582    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8790a27-ef61-43db-803b-fc295959b33a-istiod-ca-cert" pod "c8790a27-ef61-43db-803b-fc295959b33a" (UID: "c8790a27-ef61-43db-803b-fc295959b33a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:27.195634 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195628    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-socket\") pod \"c8790a27-ef61-43db-803b-fc295959b33a\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") "
Oct 29 23:03:27.195932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195663    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-data\") pod \"c8790a27-ef61-43db-803b-fc295959b33a\" (UID: \"c8790a27-ef61-43db-803b-fc295959b33a\") "
Oct 29 23:03:27.195932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195669    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-credential-socket" pod "c8790a27-ef61-43db-803b-fc295959b33a" (UID: "c8790a27-ef61-43db-803b-fc295959b33a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:27.195932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195902    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-socket" pod "c8790a27-ef61-43db-803b-fc295959b33a" (UID: "c8790a27-ef61-43db-803b-fc295959b33a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:27.195932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.195917    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-certs" pod "c8790a27-ef61-43db-803b-fc295959b33a" (UID: "c8790a27-ef61-43db-803b-fc295959b33a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:27.196158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.196109    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-data" pod "c8790a27-ef61-43db-803b-fc295959b33a" (UID: "c8790a27-ef61-43db-803b-fc295959b33a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:27.196345 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.196303    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.196487 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.196347    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.196487 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.196364    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.196487 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.196377    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8790a27-ef61-43db-803b-fc295959b33a-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.196487 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.196391    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.197714 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.197687    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-istio-token" pod "c8790a27-ef61-43db-803b-fc295959b33a" (UID: "c8790a27-ef61-43db-803b-fc295959b33a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:27.197827 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.197759    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-envoy" pod "c8790a27-ef61-43db-803b-fc295959b33a" (UID: "c8790a27-ef61-43db-803b-fc295959b33a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:27.197827 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.197782    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8790a27-ef61-43db-803b-fc295959b33a-istio-podinfo" pod "c8790a27-ef61-43db-803b-fc295959b33a" (UID: "c8790a27-ef61-43db-803b-fc295959b33a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:27.198018 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.198002    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-kube-api-access-q6vlm" pod "c8790a27-ef61-43db-803b-fc295959b33a" (UID: "c8790a27-ef61-43db-803b-fc295959b33a"). InnerVolumeSpecName "kube-api-access-q6vlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:27.297818 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.297793    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8790a27-ef61-43db-803b-fc295959b33a-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.297818 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.297816    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8790a27-ef61-43db-803b-fc295959b33a-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.297958 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.297825    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6vlm\" (UniqueName: \"kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-kube-api-access-q6vlm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.297958 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.297835    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8790a27-ef61-43db-803b-fc295959b33a-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:27.823869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.823837    4860 generic.go:356] "Generic (PLEG): container finished" podID="c8790a27-ef61-43db-803b-fc295959b33a" containerID="7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646" exitCode=0
Oct 29 23:03:27.824309 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.823886    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"
Oct 29 23:03:27.824309 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.823917    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" event={"ID":"c8790a27-ef61-43db-803b-fc295959b33a","Type":"ContainerDied","Data":"7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646"}
Oct 29 23:03:27.824309 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.823970    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59" event={"ID":"c8790a27-ef61-43db-803b-fc295959b33a","Type":"ContainerDied","Data":"18b8fb915f32b6a0d91e6ce22f42c42f4d34ee63855ea50ffa457690f420d1ed"}
Oct 29 23:03:27.824309 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.823991    4860 scope.go:122] "RemoveContainer" containerID="7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646"
Oct 29 23:03:27.835770 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.835744    4860 scope.go:122] "RemoveContainer" containerID="8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72"
Oct 29 23:03:27.845840 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.845825    4860 scope.go:122] "RemoveContainer" containerID="7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646"
Oct 29 23:03:27.846115 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:27.846097    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646\": container with ID starting with 7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646 not found: ID does not exist" containerID="7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646"
Oct 29 23:03:27.846170 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.846122    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646"} err="failed to get container status \"7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646\": rpc error: code = NotFound desc = could not find container \"7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646\": container with ID starting with 7fb134bda84fe832648f6925bf6cfaf9640a2097ca12a12ad41a321ea913f646 not found: ID does not exist"
Oct 29 23:03:27.846170 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.846138    4860 scope.go:122] "RemoveContainer" containerID="8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72"
Oct 29 23:03:27.846441 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:27.846420    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72\": container with ID starting with 8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72 not found: ID does not exist" containerID="8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72"
Oct 29 23:03:27.846506 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.846449    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72"} err="failed to get container status \"8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72\": rpc error: code = NotFound desc = could not find container \"8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72\": container with ID starting with 8835aadf112749ae736f1659d370b8c9226ab51668e851f08917a22a7c7a8f72 not found: ID does not exist"
Oct 29 23:03:27.898735 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.898703    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"]
Oct 29 23:03:27.911772 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:27.911746    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-9aafe6-openshift-default-6c9bf5dd85-fhq59"]
Oct 29 23:03:28.111414 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:28.111307    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8790a27-ef61-43db-803b-fc295959b33a" path="/var/lib/kubelet/pods/c8790a27-ef61-43db-803b-fc295959b33a/volumes"
Oct 29 23:03:28.348905 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:28.348873    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:29.670051 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670016    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"]
Oct 29 23:03:29.670537 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670521    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy"
Oct 29 23:03:29.670584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670538    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy"
Oct 29 23:03:29.670584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670563    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy"
Oct 29 23:03:29.670584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670569    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy"
Oct 29 23:03:29.670584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670576    4860 cpu_manager.go:414] "containerMap: removing container" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy"
Oct 29 23:03:29.670584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670582    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy"
Oct 29 23:03:29.670764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670602    4860 cpu_manager.go:414] "containerMap: removing container" podUID="30d6699e-f439-44ca-879b-4bb1267cd5d3" containerName="payload-pre-processing"
Oct 29 23:03:29.670764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670606    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d6699e-f439-44ca-879b-4bb1267cd5d3" containerName="payload-pre-processing"
Oct 29 23:03:29.670764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670613    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6da5e98e-cace-47e5-85b9-49142092cb0b" containerName="revoke-keys"
Oct 29 23:03:29.670764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670618    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da5e98e-cace-47e5-85b9-49142092cb0b" containerName="revoke-keys"
Oct 29 23:03:29.670764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670630    4860 cpu_manager.go:414] "containerMap: removing container" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy"
Oct 29 23:03:29.670764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670634    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy"
Oct 29 23:03:29.670764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670729    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy"
Oct 29 23:03:29.670764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670738    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8790a27-ef61-43db-803b-fc295959b33a" containerName="istio-proxy"
Oct 29 23:03:29.670764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670743    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy"
Oct 29 23:03:29.670764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670751    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d6699e-f439-44ca-879b-4bb1267cd5d3" containerName="payload-pre-processing"
Oct 29 23:03:29.670764 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.670756    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da5e98e-cace-47e5-85b9-49142092cb0b" containerName="revoke-keys"
Oct 29 23:03:29.675484 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.675466    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:29.692212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.692188    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"]
Oct 29 23:03:29.692955 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.692933    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="33379dd5-161b-4506-9860-228572c35c00" containerName="istio-proxy"
Oct 29 23:03:29.697169 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.697045    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:29.701223 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.701197    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"]
Oct 29 23:03:29.701983 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.701956    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:03:29.702766 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.702748    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:03:29.717176 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.717146    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"]
Oct 29 23:03:29.721972 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.721948    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d69a2ad3-mt6qh\" (UID: \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:29.722072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.722053    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g5gt\" (UniqueName: \"kubernetes.io/projected/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-kube-api-access-7g5gt\") pod \"maas-api-revoke-sub-d69a2ad3-mt6qh\" (UID: \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:29.722116 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.722078    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d69a2ad3-mt6qh\" (UID: \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:29.822732 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.822707    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d69a2ad3-mt6qh\" (UID: \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:29.822869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.822755    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6rbl\" (UniqueName: \"kubernetes.io/projected/f67159e2-0a49-41d4-90d1-fc3626f2c960-kube-api-access-d6rbl\") pod \"maas-api-revoke-sub-4c436615-sr52r\" (UID: \"f67159e2-0a49-41d4-90d1-fc3626f2c960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:29.822869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.822822    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g5gt\" (UniqueName: \"kubernetes.io/projected/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-kube-api-access-7g5gt\") pod \"maas-api-revoke-sub-d69a2ad3-mt6qh\" (UID: \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:29.822869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.822846    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c436615-sr52r\" (UID: \"f67159e2-0a49-41d4-90d1-fc3626f2c960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:29.822982 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.822921    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d69a2ad3-mt6qh\" (UID: \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:29.823033 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.822993    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c436615-sr52r\" (UID: \"f67159e2-0a49-41d4-90d1-fc3626f2c960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:29.823516 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.823490    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d69a2ad3-mt6qh\" (UID: \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:29.825441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.825421    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d69a2ad3-mt6qh\" (UID: \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:29.842962 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.842937    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g5gt\" (UniqueName: \"kubernetes.io/projected/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-kube-api-access-7g5gt\") pod \"maas-api-revoke-sub-d69a2ad3-mt6qh\" (UID: \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:29.924343 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.924280    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c436615-sr52r\" (UID: \"f67159e2-0a49-41d4-90d1-fc3626f2c960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:29.924462 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.924441    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c436615-sr52r\" (UID: \"f67159e2-0a49-41d4-90d1-fc3626f2c960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:29.924563 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.924550    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6rbl\" (UniqueName: \"kubernetes.io/projected/f67159e2-0a49-41d4-90d1-fc3626f2c960-kube-api-access-d6rbl\") pod \"maas-api-revoke-sub-4c436615-sr52r\" (UID: \"f67159e2-0a49-41d4-90d1-fc3626f2c960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:29.924868 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.924844    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c436615-sr52r\" (UID: \"f67159e2-0a49-41d4-90d1-fc3626f2c960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:29.926609 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.926593    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c436615-sr52r\" (UID: \"f67159e2-0a49-41d4-90d1-fc3626f2c960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:29.944536 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.944516    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6rbl\" (UniqueName: \"kubernetes.io/projected/f67159e2-0a49-41d4-90d1-fc3626f2c960-kube-api-access-d6rbl\") pod \"maas-api-revoke-sub-4c436615-sr52r\" (UID: \"f67159e2-0a49-41d4-90d1-fc3626f2c960\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:29.990546 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:29.990523    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:30.019480 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:30.019453    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:30.090884 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:30.087028    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:30.306899 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:03:30.306867    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e30c8e_fc84_4be0_8a3d_f2e01ac421e6.slice/crio-090781f059e41a044b6e3592df9c99f95e1ff1fe2ea6c444ceeb40ce22a33277 WatchSource:0}: Error finding container 090781f059e41a044b6e3592df9c99f95e1ff1fe2ea6c444ceeb40ce22a33277: Status 404 returned error can't find the container with id 090781f059e41a044b6e3592df9c99f95e1ff1fe2ea6c444ceeb40ce22a33277
Oct 29 23:03:30.309672 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:30.309645    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"]
Oct 29 23:03:30.375302 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:03:30.375276    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67159e2_0a49_41d4_90d1_fc3626f2c960.slice/crio-f98a528fd580e56b5ca829653369d1d2695cb47de78e8306a6e534bff9008fdc WatchSource:0}: Error finding container f98a528fd580e56b5ca829653369d1d2695cb47de78e8306a6e534bff9008fdc: Status 404 returned error can't find the container with id f98a528fd580e56b5ca829653369d1d2695cb47de78e8306a6e534bff9008fdc
Oct 29 23:03:30.378719 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:30.378699    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"]
Oct 29 23:03:30.507973 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:30.507933    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv" podUID="5962c518-2366-4739-995f-0caa8149b9be" containerName="payload-processing" containerID="cri-o://bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2" gracePeriod=25
Oct 29 23:03:30.745334 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:30.745301    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg" podUID="646e1a1e-8f19-4df7-91d8-4864270cae7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:30.849010 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:30.848945    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh" event={"ID":"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6","Type":"ContainerStarted","Data":"6e4e1e77415cc5f166c6fa5b12757e18256cb23efb6e185412e1467e2e680b55"}
Oct 29 23:03:30.849010 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:30.849009    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh" event={"ID":"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6","Type":"ContainerStarted","Data":"090781f059e41a044b6e3592df9c99f95e1ff1fe2ea6c444ceeb40ce22a33277"}
Oct 29 23:03:30.850763 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:30.850738    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r" event={"ID":"f67159e2-0a49-41d4-90d1-fc3626f2c960","Type":"ContainerStarted","Data":"22a3a40418f047402804b93b34cda6029bff7ce97d20db6e0dd74f14c7b18147"}
Oct 29 23:03:30.850881 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:30.850780    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r" event={"ID":"f67159e2-0a49-41d4-90d1-fc3626f2c960","Type":"ContainerStarted","Data":"f98a528fd580e56b5ca829653369d1d2695cb47de78e8306a6e534bff9008fdc"}
Oct 29 23:03:30.894681 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:30.894630    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh" podStartSLOduration=1.894614985 podStartE2EDuration="1.894614985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:30.894565706 +0000 UTC m=+2952.919341862" watchObservedRunningTime="2026-10-29 23:03:30.894614985 +0000 UTC m=+2952.919391118"
Oct 29 23:03:30.937638 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:30.937598    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r" podStartSLOduration=1.93758636 podStartE2EDuration="1.93758636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:30.934540807 +0000 UTC m=+2952.959316939" watchObservedRunningTime="2026-10-29 23:03:30.93758636 +0000 UTC m=+2952.962362491"
Oct 29 23:03:31.859958 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:31.859929    4860 generic.go:356] "Generic (PLEG): container finished" podID="11e30c8e-fc84-4be0-8a3d-f2e01ac421e6" containerID="6e4e1e77415cc5f166c6fa5b12757e18256cb23efb6e185412e1467e2e680b55" exitCode=0
Oct 29 23:03:31.860354 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:31.860001    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh" event={"ID":"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6","Type":"ContainerDied","Data":"6e4e1e77415cc5f166c6fa5b12757e18256cb23efb6e185412e1467e2e680b55"}
Oct 29 23:03:31.861684 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:31.861649    4860 generic.go:356] "Generic (PLEG): container finished" podID="f67159e2-0a49-41d4-90d1-fc3626f2c960" containerID="22a3a40418f047402804b93b34cda6029bff7ce97d20db6e0dd74f14c7b18147" exitCode=0
Oct 29 23:03:31.861817 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:31.861727    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r" event={"ID":"f67159e2-0a49-41d4-90d1-fc3626f2c960","Type":"ContainerDied","Data":"22a3a40418f047402804b93b34cda6029bff7ce97d20db6e0dd74f14c7b18147"}
Oct 29 23:03:32.203072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:32.203042    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:32.933980 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:32.933949    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:33.058051 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.058006    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:33.063837 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.063813    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:33.109802 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.109774    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:33.163835 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.163763    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-kube-api-access-7g5gt\" (UniqueName: \"kubernetes.io/projected/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-kube-api-access-7g5gt\") pod \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\" (UID: \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\") "
Oct 29 23:03:33.163835 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.163807    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-openshift-ca\") pod \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\" (UID: \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\") "
Oct 29 23:03:33.164211 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.163875    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-xks-ca\") pod \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\" (UID: \"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6\") "
Oct 29 23:03:33.164211 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.164037    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67159e2-0a49-41d4-90d1-fc3626f2c960-kube-api-access-d6rbl\" (UniqueName: \"kubernetes.io/projected/f67159e2-0a49-41d4-90d1-fc3626f2c960-kube-api-access-d6rbl\") pod \"f67159e2-0a49-41d4-90d1-fc3626f2c960\" (UID: \"f67159e2-0a49-41d4-90d1-fc3626f2c960\") "
Oct 29 23:03:33.164211 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.164092    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-openshift-ca\") pod \"f67159e2-0a49-41d4-90d1-fc3626f2c960\" (UID: \"f67159e2-0a49-41d4-90d1-fc3626f2c960\") "
Oct 29 23:03:33.164440 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.164239    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-openshift-ca" pod "11e30c8e-fc84-4be0-8a3d-f2e01ac421e6" (UID: "11e30c8e-fc84-4be0-8a3d-f2e01ac421e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:33.164498 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.164444    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-xks-ca\") pod \"f67159e2-0a49-41d4-90d1-fc3626f2c960\" (UID: \"f67159e2-0a49-41d4-90d1-fc3626f2c960\") "
Oct 29 23:03:33.164498 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.164484    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-openshift-ca" pod "f67159e2-0a49-41d4-90d1-fc3626f2c960" (UID: "f67159e2-0a49-41d4-90d1-fc3626f2c960"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:33.165463 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.165410    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:33.165656 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.165633    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:33.166762 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.166680    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-kube-api-access-7g5gt" pod "11e30c8e-fc84-4be0-8a3d-f2e01ac421e6" (UID: "11e30c8e-fc84-4be0-8a3d-f2e01ac421e6"). InnerVolumeSpecName "kube-api-access-7g5gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:33.166893 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.166789    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67159e2-0a49-41d4-90d1-fc3626f2c960-kube-api-access-d6rbl" pod "f67159e2-0a49-41d4-90d1-fc3626f2c960" (UID: "f67159e2-0a49-41d4-90d1-fc3626f2c960"). InnerVolumeSpecName "kube-api-access-d6rbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:33.166893 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.166793    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-xks-ca" pod "11e30c8e-fc84-4be0-8a3d-f2e01ac421e6" (UID: "11e30c8e-fc84-4be0-8a3d-f2e01ac421e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:33.167590 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.167570    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-xks-ca" pod "f67159e2-0a49-41d4-90d1-fc3626f2c960" (UID: "f67159e2-0a49-41d4-90d1-fc3626f2c960"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:33.267275 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.267251    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:33.267275 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.267276    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6rbl\" (UniqueName: \"kubernetes.io/projected/f67159e2-0a49-41d4-90d1-fc3626f2c960-kube-api-access-d6rbl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:33.267448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.267286    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67159e2-0a49-41d4-90d1-fc3626f2c960-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:33.267448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.267295    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g5gt\" (UniqueName: \"kubernetes.io/projected/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6-kube-api-access-7g5gt\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:33.733491 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.733458    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:33.887444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.887413    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"
Oct 29 23:03:33.887641 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.887453    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r" event={"ID":"f67159e2-0a49-41d4-90d1-fc3626f2c960","Type":"ContainerDied","Data":"f98a528fd580e56b5ca829653369d1d2695cb47de78e8306a6e534bff9008fdc"}
Oct 29 23:03:33.887641 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.887543    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f98a528fd580e56b5ca829653369d1d2695cb47de78e8306a6e534bff9008fdc"
Oct 29 23:03:33.889086 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.889064    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"
Oct 29 23:03:33.889218 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.889100    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh" event={"ID":"11e30c8e-fc84-4be0-8a3d-f2e01ac421e6","Type":"ContainerDied","Data":"090781f059e41a044b6e3592df9c99f95e1ff1fe2ea6c444ceeb40ce22a33277"}
Oct 29 23:03:33.889218 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:33.889141    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="090781f059e41a044b6e3592df9c99f95e1ff1fe2ea6c444ceeb40ce22a33277"
Oct 29 23:03:34.028389 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:34.028301    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"]
Oct 29 23:03:34.031288 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:34.031265    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ac04dd-ng2ld"]
Oct 29 23:03:34.113603 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:34.113569    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3860308b-e709-4dc4-96ad-875f5c15c551" path="/var/lib/kubelet/pods/3860308b-e709-4dc4-96ad-875f5c15c551/volumes"
Oct 29 23:03:34.857722 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:34.857662    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" containerID="cri-o://c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e" gracePeriod=45
Oct 29 23:03:34.873783 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:34.873753    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main" containerID="cri-o://0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0" gracePeriod=45
Oct 29 23:03:35.160811 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.160784    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 23:03:35.232739 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.232706    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 23:03:35.256722 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.256690    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:35.289442 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.289413    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-kube-api-access-wtwlw\" (UniqueName: \"kubernetes.io/projected/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-kube-api-access-wtwlw\") pod \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") "
Oct 29 23:03:35.289609 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.289457    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-home\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-home\") pod \"482698c2-d44a-4564-b588-d402fd9cac23\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") "
Oct 29 23:03:35.289609 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.289582    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-dshm\") pod \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") "
Oct 29 23:03:35.289729 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.289625    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-home\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-home\") pod \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") "
Oct 29 23:03:35.289729 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.289668    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-model-cache\") pod \"482698c2-d44a-4564-b588-d402fd9cac23\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") "
Oct 29 23:03:35.289729 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.289705    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tls-certs\") pod \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") "
Oct 29 23:03:35.289908 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.289746    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-home" pod "482698c2-d44a-4564-b588-d402fd9cac23" (UID: "482698c2-d44a-4564-b588-d402fd9cac23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.289908 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.289762    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-model-cache\") pod \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") "
Oct 29 23:03:35.289908 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.289841    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tmp-dir\") pod \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\" (UID: \"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735\") "
Oct 29 23:03:35.290095 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.289924    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-home" pod "6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" (UID: "6d99f80b-82a0-4f37-b1e2-8d85a6ba1735"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.290095 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.289931    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-model-cache" pod "482698c2-d44a-4564-b588-d402fd9cac23" (UID: "482698c2-d44a-4564-b588-d402fd9cac23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.290218 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.290108    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-model-cache" pod "6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" (UID: "6d99f80b-82a0-4f37-b1e2-8d85a6ba1735"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.290218 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.290206    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tmp-dir" pod "6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" (UID: "6d99f80b-82a0-4f37-b1e2-8d85a6ba1735"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.290349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.290272    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/482698c2-d44a-4564-b588-d402fd9cac23-tls-certs\" (UniqueName: \"kubernetes.io/secret/482698c2-d44a-4564-b588-d402fd9cac23-tls-certs\") pod \"482698c2-d44a-4564-b588-d402fd9cac23\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") "
Oct 29 23:03:35.290417 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.290360    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-tmp-dir\") pod \"482698c2-d44a-4564-b588-d402fd9cac23\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") "
Oct 29 23:03:35.290417 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.290390    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-dshm\") pod \"482698c2-d44a-4564-b588-d402fd9cac23\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") "
Oct 29 23:03:35.290522 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.290456    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482698c2-d44a-4564-b588-d402fd9cac23-kube-api-access-t6f7s\" (UniqueName: \"kubernetes.io/projected/482698c2-d44a-4564-b588-d402fd9cac23-kube-api-access-t6f7s\") pod \"482698c2-d44a-4564-b588-d402fd9cac23\" (UID: \"482698c2-d44a-4564-b588-d402fd9cac23\") "
Oct 29 23:03:35.291646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.290850    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-tmp-dir" pod "482698c2-d44a-4564-b588-d402fd9cac23" (UID: "482698c2-d44a-4564-b588-d402fd9cac23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.291646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.291248    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.291646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.291271    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.291646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.291284    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.291646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.291296    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.291646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.291310    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.291646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.291340    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.292456 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.292361    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-kube-api-access-wtwlw" pod "6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" (UID: "6d99f80b-82a0-4f37-b1e2-8d85a6ba1735"). InnerVolumeSpecName "kube-api-access-wtwlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:35.292832 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.292800    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tls-certs" pod "6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" (UID: "6d99f80b-82a0-4f37-b1e2-8d85a6ba1735"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:35.292941 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.292839    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482698c2-d44a-4564-b588-d402fd9cac23-tls-certs" pod "482698c2-d44a-4564-b588-d402fd9cac23" (UID: "482698c2-d44a-4564-b588-d402fd9cac23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:35.293916 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.293494    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482698c2-d44a-4564-b588-d402fd9cac23-kube-api-access-t6f7s" pod "482698c2-d44a-4564-b588-d402fd9cac23" (UID: "482698c2-d44a-4564-b588-d402fd9cac23"). InnerVolumeSpecName "kube-api-access-t6f7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:35.294491 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.294467    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-dshm" pod "482698c2-d44a-4564-b588-d402fd9cac23" (UID: "482698c2-d44a-4564-b588-d402fd9cac23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.294669 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.294646    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-dshm" pod "6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" (UID: "6d99f80b-82a0-4f37-b1e2-8d85a6ba1735"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:35.395534 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.394600    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/482698c2-d44a-4564-b588-d402fd9cac23-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.395534 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.394707    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/482698c2-d44a-4564-b588-d402fd9cac23-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.395534 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.394721    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6f7s\" (UniqueName: \"kubernetes.io/projected/482698c2-d44a-4564-b588-d402fd9cac23-kube-api-access-t6f7s\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.395534 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.394734    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtwlw\" (UniqueName: \"kubernetes.io/projected/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-kube-api-access-wtwlw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.395534 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.394747    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.395534 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.394760    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:35.904258 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.904226    4860 generic.go:356] "Generic (PLEG): container finished" podID="482698c2-d44a-4564-b588-d402fd9cac23" containerID="c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e" exitCode=0
Oct 29 23:03:35.904477 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.904271    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"
Oct 29 23:03:35.904477 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.904353    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" event={"ID":"482698c2-d44a-4564-b588-d402fd9cac23","Type":"ContainerDied","Data":"c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e"}
Oct 29 23:03:35.904477 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.904399    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" event={"ID":"482698c2-d44a-4564-b588-d402fd9cac23","Type":"ContainerDied","Data":"877f1b7a1e958a2dcccf8177a006d64cfc7b1a524cff4455213301740538435e"}
Oct 29 23:03:35.904477 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.904421    4860 scope.go:122] "RemoveContainer" containerID="c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e"
Oct 29 23:03:35.906075 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.906044    4860 generic.go:356] "Generic (PLEG): container finished" podID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerID="0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0" exitCode=0
Oct 29 23:03:35.906186 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.906102    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"
Oct 29 23:03:35.906186 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.906133    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" event={"ID":"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735","Type":"ContainerDied","Data":"0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0"}
Oct 29 23:03:35.906186 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.906171    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh" event={"ID":"6d99f80b-82a0-4f37-b1e2-8d85a6ba1735","Type":"ContainerDied","Data":"874e83521460b2f4d4b55296a03c4e218ff0ca9dfba380dc6f3fa30eeadd8432"}
Oct 29 23:03:35.916647 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.916631    4860 scope.go:122] "RemoveContainer" containerID="c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e"
Oct 29 23:03:35.916889 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:35.916871    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e\": container with ID starting with c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e not found: ID does not exist" containerID="c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e"
Oct 29 23:03:35.916935 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.916896    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e"} err="failed to get container status \"c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e\": rpc error: code = NotFound desc = could not find container \"c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e\": container with ID starting with c60436016aed4007e31496b6b0ccc3fb6eeff736c2192a115313e92f1dce435e not found: ID does not exist"
Oct 29 23:03:35.916935 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.916911    4860 scope.go:122] "RemoveContainer" containerID="0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0"
Oct 29 23:03:35.926919 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.926895    4860 scope.go:122] "RemoveContainer" containerID="0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0"
Oct 29 23:03:35.927170 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:35.927149    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0\": container with ID starting with 0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0 not found: ID does not exist" containerID="0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0"
Oct 29 23:03:35.927225 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.927175    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0"} err="failed to get container status \"0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0\": rpc error: code = NotFound desc = could not find container \"0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0\": container with ID starting with 0232e2392eb36c4f4f1ef733903558c2d24d84ce7803d57dda6a5ced78e5a5f0 not found: ID does not exist"
Oct 29 23:03:35.959069 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.959046    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"]
Oct 29 23:03:35.972622 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.972598    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp"]
Oct 29 23:03:35.999916 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:35.999888    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"]
Oct 29 23:03:36.003163 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:36.003139    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2ba10a/premium-w2-2ba10a-kserve-79f58b59f9-cf5bh"]
Oct 29 23:03:36.108298 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:36.108269    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2ba10a/distinct-2-w2-2ba10a-kserve-c595cb7f7-n9sqp" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:03:36.110949 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:36.110920    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482698c2-d44a-4564-b588-d402fd9cac23" path="/var/lib/kubelet/pods/482698c2-d44a-4564-b588-d402fd9cac23/volumes"
Oct 29 23:03:36.111439 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:36.111421    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" path="/var/lib/kubelet/pods/6d99f80b-82a0-4f37-b1e2-8d85a6ba1735/volumes"
Oct 29 23:03:36.605108 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:36.605070    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:36.695451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:36.695414    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv" podUID="5962c518-2366-4739-995f-0caa8149b9be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:36.752228 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:36.752197    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:36.860858 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:36.860788    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:37.260601 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:37.260568    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:37.369022 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:37.368994    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"]
Oct 29 23:03:37.382260 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:37.382231    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-80248b-m7r9w"]
Oct 29 23:03:37.385842 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:37.385814    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:37.498637 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:37.498602    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:37.858629 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:37.858597    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:38.113399 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:38.113310    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da5e98e-cace-47e5-85b9-49142092cb0b" path="/var/lib/kubelet/pods/6da5e98e-cace-47e5-85b9-49142092cb0b/volumes"
Oct 29 23:03:39.146771 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:39.146730    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:40.745074 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:40.745039    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:40.745552 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:40.745198    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg" podUID="646e1a1e-8f19-4df7-91d8-4864270cae7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:41.146555 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.146530    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"
Oct 29 23:03:41.273295 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.273261    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646e1a1e-8f19-4df7-91d8-4864270cae7d-kube-api-access-qjvcl\" (UniqueName: \"kubernetes.io/projected/646e1a1e-8f19-4df7-91d8-4864270cae7d-kube-api-access-qjvcl\") pod \"646e1a1e-8f19-4df7-91d8-4864270cae7d\" (UID: \"646e1a1e-8f19-4df7-91d8-4864270cae7d\") "
Oct 29 23:03:41.273526 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.273469    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/646e1a1e-8f19-4df7-91d8-4864270cae7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646e1a1e-8f19-4df7-91d8-4864270cae7d-plugins-config-volume\") pod \"646e1a1e-8f19-4df7-91d8-4864270cae7d\" (UID: \"646e1a1e-8f19-4df7-91d8-4864270cae7d\") "
Oct 29 23:03:41.273835 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.273811    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646e1a1e-8f19-4df7-91d8-4864270cae7d-plugins-config-volume" pod "646e1a1e-8f19-4df7-91d8-4864270cae7d" (UID: "646e1a1e-8f19-4df7-91d8-4864270cae7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:41.275513 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.275490    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646e1a1e-8f19-4df7-91d8-4864270cae7d-kube-api-access-qjvcl" pod "646e1a1e-8f19-4df7-91d8-4864270cae7d" (UID: "646e1a1e-8f19-4df7-91d8-4864270cae7d"). InnerVolumeSpecName "kube-api-access-qjvcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:41.374614 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.374561    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646e1a1e-8f19-4df7-91d8-4864270cae7d-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.374614 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.374583    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjvcl\" (UniqueName: \"kubernetes.io/projected/646e1a1e-8f19-4df7-91d8-4864270cae7d-kube-api-access-qjvcl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:41.951705 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.951671    4860 generic.go:356] "Generic (PLEG): container finished" podID="646e1a1e-8f19-4df7-91d8-4864270cae7d" containerID="f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5" exitCode=0
Oct 29 23:03:41.952158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.951716    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"
Oct 29 23:03:41.952158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.951734    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg" event={"ID":"646e1a1e-8f19-4df7-91d8-4864270cae7d","Type":"ContainerDied","Data":"f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5"}
Oct 29 23:03:41.952158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.951787    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg" event={"ID":"646e1a1e-8f19-4df7-91d8-4864270cae7d","Type":"ContainerDied","Data":"7855383ab026276716bf881dcfecb73867695f0419d42980baa731bdbaef3dc9"}
Oct 29 23:03:41.952158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.951818    4860 scope.go:122] "RemoveContainer" containerID="f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5"
Oct 29 23:03:41.963060 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.963041    4860 scope.go:122] "RemoveContainer" containerID="f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5"
Oct 29 23:03:41.963312 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:41.963292    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5\": container with ID starting with f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5 not found: ID does not exist" containerID="f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5"
Oct 29 23:03:41.963409 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:41.963350    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5"} err="failed to get container status \"f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5\": rpc error: code = NotFound desc = could not find container \"f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5\": container with ID starting with f7b8ffd75d9e12a6f4cfc27cb0ba5cebb3801c2173e8ebed841c5295941d59b5 not found: ID does not exist"
Oct 29 23:03:42.007446 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:42.007420    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"]
Oct 29 23:03:42.019989 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:42.019969    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2e046a-6888946c9d-w2lbg"]
Oct 29 23:03:42.112216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:42.112193    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="646e1a1e-8f19-4df7-91d8-4864270cae7d" path="/var/lib/kubelet/pods/646e1a1e-8f19-4df7-91d8-4864270cae7d/volumes"
Oct 29 23:03:42.374534 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:42.374458    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:43.972200 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:43.972169    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:44.095497 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:44.095451    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:44.217990 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:44.217918    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:44.337876 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:44.337792    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:44.509497 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:44.509463    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:44.627588 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:44.627511    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:44.764687 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:44.764653    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:45.031849 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:45.031816    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"]
Oct 29 23:03:45.044187 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:45.044158    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7dc7eff-7rnvs"]
Oct 29 23:03:45.053112 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:45.053088    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:45.174672 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:45.174636    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:45.194441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:45.194409    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"]
Oct 29 23:03:45.199892 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:45.199855    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2e046a-pm8nx"]
Oct 29 23:03:45.298075 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:45.298001    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:45.698531 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:45.698497    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:46.116198 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:46.116112    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8533ae55-d0b6-4885-9c79-14416574cf0d" path="/var/lib/kubelet/pods/8533ae55-d0b6-4885-9c79-14416574cf0d/volumes"
Oct 29 23:03:46.116779 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:46.116758    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96fea28-eee8-4c7d-8c3b-435d81aa80f6" path="/var/lib/kubelet/pods/f96fea28-eee8-4c7d-8c3b-435d81aa80f6/volumes"
Oct 29 23:03:46.618689 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:46.618655    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:46.695642 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:46.695601    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv" podUID="5962c518-2366-4739-995f-0caa8149b9be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:03:46.736304 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:46.736274    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:47.358054 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:47.358021    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:47.519309 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:47.519274    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:47.924253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:47.924199    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:48.046710 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:48.046670    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:48.056239 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:48.056207    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:03:48.406141 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:48.406105    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:49.273722 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:49.273678    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:50.771580 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:50.771523    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"
Oct 29 23:03:50.800084 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:50.800058    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5962c518-2366-4739-995f-0caa8149b9be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5962c518-2366-4739-995f-0caa8149b9be-plugins-config-volume\") pod \"5962c518-2366-4739-995f-0caa8149b9be\" (UID: \"5962c518-2366-4739-995f-0caa8149b9be\") "
Oct 29 23:03:50.800193 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:50.800106    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5962c518-2366-4739-995f-0caa8149b9be-kube-api-access-z4fp8\" (UniqueName: \"kubernetes.io/projected/5962c518-2366-4739-995f-0caa8149b9be-kube-api-access-z4fp8\") pod \"5962c518-2366-4739-995f-0caa8149b9be\" (UID: \"5962c518-2366-4739-995f-0caa8149b9be\") "
Oct 29 23:03:50.800405 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:50.800384    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5962c518-2366-4739-995f-0caa8149b9be-plugins-config-volume" pod "5962c518-2366-4739-995f-0caa8149b9be" (UID: "5962c518-2366-4739-995f-0caa8149b9be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:50.800605 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:50.800585    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5962c518-2366-4739-995f-0caa8149b9be-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:50.802252 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:50.802233    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5962c518-2366-4739-995f-0caa8149b9be-kube-api-access-z4fp8" pod "5962c518-2366-4739-995f-0caa8149b9be" (UID: "5962c518-2366-4739-995f-0caa8149b9be"). InnerVolumeSpecName "kube-api-access-z4fp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:50.898308 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:50.898278    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:50.901609 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:50.901583    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4fp8\" (UniqueName: \"kubernetes.io/projected/5962c518-2366-4739-995f-0caa8149b9be-kube-api-access-z4fp8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:51.028727 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:51.028641    4860 generic.go:356] "Generic (PLEG): container finished" podID="5962c518-2366-4739-995f-0caa8149b9be" containerID="bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2" exitCode=0
Oct 29 23:03:51.028727 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:51.028687    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"
Oct 29 23:03:51.028980 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:51.028727    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv" event={"ID":"5962c518-2366-4739-995f-0caa8149b9be","Type":"ContainerDied","Data":"bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2"}
Oct 29 23:03:51.028980 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:51.028783    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv" event={"ID":"5962c518-2366-4739-995f-0caa8149b9be","Type":"ContainerDied","Data":"743398570cfa876de1d19a760105460e0561822afe4f2592d34d0bd455dc1f0d"}
Oct 29 23:03:51.028980 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:51.028814    4860 scope.go:122] "RemoveContainer" containerID="bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2"
Oct 29 23:03:51.039441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:51.039422    4860 scope.go:122] "RemoveContainer" containerID="bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2"
Oct 29 23:03:51.039701 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:51.039684    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2\": container with ID starting with bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2 not found: ID does not exist" containerID="bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2"
Oct 29 23:03:51.039753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:51.039710    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2"} err="failed to get container status \"bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2\": rpc error: code = NotFound desc = could not find container \"bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2\": container with ID starting with bc021d03aa402884e8c773ccbcd0b61aa6c087d05b2571ce87c3da9064c702d2 not found: ID does not exist"
Oct 29 23:03:51.084028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:51.084004    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"]
Oct 29 23:03:51.096552 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:51.096533    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2ba10a-78fb8d6fd4-qw5lv"]
Oct 29 23:03:51.822361 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:51.822319    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:51.949163 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:51.949123    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:52.121762 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:52.121667    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5962c518-2366-4739-995f-0caa8149b9be" path="/var/lib/kubelet/pods/5962c518-2366-4739-995f-0caa8149b9be/volumes"
Oct 29 23:03:52.184318 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:52.184284    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:52.332447 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:52.332414    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:52.467578 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:52.467547    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:52.601753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:52.601694    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:52.623974 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:52.623943    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"]
Oct 29 23:03:52.624217 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:52.624189    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" podUID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" containerName="istio-proxy" containerID="cri-o://7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731" gracePeriod=30
Oct 29 23:03:52.725946 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:52.725867    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:54.363197 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:54.363165    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:54.489838 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:54.489809    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:54.630916 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:54.630804    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:54.809544 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:54.809509    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:56.560836 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:56.560802    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:56.915994 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:56.915894    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:57.078704 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.078667    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:57.265773 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.265743    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:57.438098 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.438060    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:57.588541 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.588468    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"]
Oct 29 23:03:57.693030 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.692997    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:57.887647 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.887621    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 23:03:57.974106 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.974039    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:57.983201 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.983172    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-kube-api-access-fbkcw\" (UniqueName: \"kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-kube-api-access-fbkcw\") pod \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") "
Oct 29 23:03:57.983387 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.983218    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istiod-ca-cert\") pod \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") "
Oct 29 23:03:57.983387 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.983298    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-token\" (UniqueName: \"kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-token\") pod \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") "
Oct 29 23:03:57.983387 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.983364    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-credential-socket\") pod \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") "
Oct 29 23:03:57.983664 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.983420    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-socket\") pod \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") "
Oct 29 23:03:57.983664 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.983477    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-envoy\") pod \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") "
Oct 29 23:03:57.983664 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.983508    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-data\") pod \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") "
Oct 29 23:03:57.983664 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.983570    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-podinfo\") pod \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") "
Oct 29 23:03:57.983664 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.983604    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-certs\") pod \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\" (UID: \"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5\") "
Oct 29 23:03:57.983899 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.983775    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-credential-socket" pod "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" (UID: "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.984202 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.983997    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-certs" pod "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" (UID: "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.984202 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.984191    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.984381 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.984216    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:57.984381 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.984282    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-data" pod "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" (UID: "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.984491 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.984396    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-socket" pod "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" (UID: "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.984755 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.984707    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istiod-ca-cert" pod "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" (UID: "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:57.985969 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.985944    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-kube-api-access-fbkcw" pod "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" (UID: "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5"). InnerVolumeSpecName "kube-api-access-fbkcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:57.986371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.986320    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-envoy" pod "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" (UID: "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:03:57.986490 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.986409    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-podinfo" pod "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" (UID: "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:03:57.986490 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:57.986458    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-token" pod "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" (UID: "ddb9bfa7-8e72-4849-8ef1-b00c24761ba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:58.084818 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.084787    4860 generic.go:356] "Generic (PLEG): container finished" podID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" containerID="7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731" exitCode=0
Oct 29 23:03:58.084955 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.084837    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"
Oct 29 23:03:58.084955 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.084873    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" event={"ID":"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5","Type":"ContainerDied","Data":"7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731"}
Oct 29 23:03:58.084955 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.084928    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4" event={"ID":"ddb9bfa7-8e72-4849-8ef1-b00c24761ba5","Type":"ContainerDied","Data":"2ff851f4d624ed0b626e99f0e8fdf62df3acafcf07947eeaff40dddb041d1aa0"}
Oct 29 23:03:58.084955 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.084951    4860 scope.go:122] "RemoveContainer" containerID="7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731"
Oct 29 23:03:58.085160 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.085045    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.085230 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.085193    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.085280 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.085230    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.085280 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.085261    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.085280 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.085275    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.085433 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.085288    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbkcw\" (UniqueName: \"kubernetes.io/projected/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-kube-api-access-fbkcw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.085433 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.085303    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:03:58.096375 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.096317    4860 scope.go:122] "RemoveContainer" containerID="7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731"
Oct 29 23:03:58.096672 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:03:58.096654    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731\": container with ID starting with 7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731 not found: ID does not exist" containerID="7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731"
Oct 29 23:03:58.096720 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.096679    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731"} err="failed to get container status \"7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731\": rpc error: code = NotFound desc = could not find container \"7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731\": container with ID starting with 7cad9ea0c8342e2e2670b3329ac9e381bce042ba2d4947f8f172c618dd36b731 not found: ID does not exist"
Oct 29 23:03:58.140613 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.140549    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"]
Oct 29 23:03:58.149486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.149460    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e046a-openshift-default-79c5dd5b49-rkgz4"]
Oct 29 23:03:58.310955 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:58.310906    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:59.805777 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:03:59.805740    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:00.112798 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:00.112728    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" path="/var/lib/kubelet/pods/ddb9bfa7-8e72-4849-8ef1-b00c24761ba5/volumes"
Oct 29 23:04:01.420334 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:01.420289    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:03.149085 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:03.149054    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:03.341624 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:03.341588    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"]
Oct 29 23:04:03.377218 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:03.377187    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"]
Oct 29 23:04:03.504251 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:03.504222    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:03.626300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:03.626261    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:04.133749 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:04.133716    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:04.506108 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:04.506080    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:04.669148 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:04.668538    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:04.785909 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:04.785807    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:05.040217 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:05.040145    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:05.165714 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:05.165682    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:05.473548 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:05.473520    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:05.629294 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:05.629269    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:06.746036 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:06.745988    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:08.342708 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.342669    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p" podUID="35221c46-797c-4c8e-97a5-55d9bc6b4829" containerName="payload-processing" containerID="cri-o://c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559" gracePeriod=25
Oct 29 23:04:08.378283 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.378254    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr" podUID="601556d3-a9a1-43b8-88b7-320af479646d" containerName="payload-pre-processing" containerID="cri-o://d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373" gracePeriod=25
Oct 29 23:04:08.383742 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.383717    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:08.645360 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.645226    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"
Oct 29 23:04:08.667143 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.667121    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"
Oct 29 23:04:08.701692 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.701667    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35221c46-797c-4c8e-97a5-55d9bc6b4829-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35221c46-797c-4c8e-97a5-55d9bc6b4829-plugins-config-volume\") pod \"35221c46-797c-4c8e-97a5-55d9bc6b4829\" (UID: \"35221c46-797c-4c8e-97a5-55d9bc6b4829\") "
Oct 29 23:04:08.701883 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.701869    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35221c46-797c-4c8e-97a5-55d9bc6b4829-kube-api-access-4tvbc\" (UniqueName: \"kubernetes.io/projected/35221c46-797c-4c8e-97a5-55d9bc6b4829-kube-api-access-4tvbc\") pod \"35221c46-797c-4c8e-97a5-55d9bc6b4829\" (UID: \"35221c46-797c-4c8e-97a5-55d9bc6b4829\") "
Oct 29 23:04:08.702062 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.702039    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35221c46-797c-4c8e-97a5-55d9bc6b4829-plugins-config-volume" pod "35221c46-797c-4c8e-97a5-55d9bc6b4829" (UID: "35221c46-797c-4c8e-97a5-55d9bc6b4829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:08.702215 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.702199    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35221c46-797c-4c8e-97a5-55d9bc6b4829-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:08.704034 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.704007    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35221c46-797c-4c8e-97a5-55d9bc6b4829-kube-api-access-4tvbc" pod "35221c46-797c-4c8e-97a5-55d9bc6b4829" (UID: "35221c46-797c-4c8e-97a5-55d9bc6b4829"). InnerVolumeSpecName "kube-api-access-4tvbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:08.803260 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.803227    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/601556d3-a9a1-43b8-88b7-320af479646d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/601556d3-a9a1-43b8-88b7-320af479646d-plugins-config-volume\") pod \"601556d3-a9a1-43b8-88b7-320af479646d\" (UID: \"601556d3-a9a1-43b8-88b7-320af479646d\") "
Oct 29 23:04:08.803462 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.803277    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/601556d3-a9a1-43b8-88b7-320af479646d-kube-api-access-96gvl\" (UniqueName: \"kubernetes.io/projected/601556d3-a9a1-43b8-88b7-320af479646d-kube-api-access-96gvl\") pod \"601556d3-a9a1-43b8-88b7-320af479646d\" (UID: \"601556d3-a9a1-43b8-88b7-320af479646d\") "
Oct 29 23:04:08.803634 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.803611    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601556d3-a9a1-43b8-88b7-320af479646d-plugins-config-volume" pod "601556d3-a9a1-43b8-88b7-320af479646d" (UID: "601556d3-a9a1-43b8-88b7-320af479646d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:08.803787 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.803773    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/601556d3-a9a1-43b8-88b7-320af479646d-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:08.803836 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.803789    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tvbc\" (UniqueName: \"kubernetes.io/projected/35221c46-797c-4c8e-97a5-55d9bc6b4829-kube-api-access-4tvbc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:08.805350 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.805311    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601556d3-a9a1-43b8-88b7-320af479646d-kube-api-access-96gvl" pod "601556d3-a9a1-43b8-88b7-320af479646d" (UID: "601556d3-a9a1-43b8-88b7-320af479646d"). InnerVolumeSpecName "kube-api-access-96gvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:08.905274 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:08.905217    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96gvl\" (UniqueName: \"kubernetes.io/projected/601556d3-a9a1-43b8-88b7-320af479646d-kube-api-access-96gvl\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:09.166535 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.166445    4860 generic.go:356] "Generic (PLEG): container finished" podID="35221c46-797c-4c8e-97a5-55d9bc6b4829" containerID="c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559" exitCode=0
Oct 29 23:04:09.166535 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.166502    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"
Oct 29 23:04:09.166740 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.166525    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p" event={"ID":"35221c46-797c-4c8e-97a5-55d9bc6b4829","Type":"ContainerDied","Data":"c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559"}
Oct 29 23:04:09.166740 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.166584    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p" event={"ID":"35221c46-797c-4c8e-97a5-55d9bc6b4829","Type":"ContainerDied","Data":"bdc455d5bddb40a408827ab8e59141173dcc2c2803219d69897430c8329853ce"}
Oct 29 23:04:09.166740 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.166600    4860 scope.go:122] "RemoveContainer" containerID="c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559"
Oct 29 23:04:09.168018 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.167983    4860 generic.go:356] "Generic (PLEG): container finished" podID="601556d3-a9a1-43b8-88b7-320af479646d" containerID="d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373" exitCode=0
Oct 29 23:04:09.168135 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.168032    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr" event={"ID":"601556d3-a9a1-43b8-88b7-320af479646d","Type":"ContainerDied","Data":"d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373"}
Oct 29 23:04:09.168135 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.168086    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr" event={"ID":"601556d3-a9a1-43b8-88b7-320af479646d","Type":"ContainerDied","Data":"50c53af52856284a9dd0e9e1c19827ec5b761c59955385a5f169f6f2ff0fe271"}
Oct 29 23:04:09.168135 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.168041    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"
Oct 29 23:04:09.178375 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.178356    4860 scope.go:122] "RemoveContainer" containerID="c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559"
Oct 29 23:04:09.178679 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:04:09.178660    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559\": container with ID starting with c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559 not found: ID does not exist" containerID="c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559"
Oct 29 23:04:09.178781 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.178691    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559"} err="failed to get container status \"c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559\": rpc error: code = NotFound desc = could not find container \"c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559\": container with ID starting with c477493e2db7f2c48ed62fcded34756667ca41c4dc9fdda45594ff873fe82559 not found: ID does not exist"
Oct 29 23:04:09.178781 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.178711    4860 scope.go:122] "RemoveContainer" containerID="d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373"
Oct 29 23:04:09.188099 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.188072    4860 scope.go:122] "RemoveContainer" containerID="d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373"
Oct 29 23:04:09.188369 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:04:09.188351    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373\": container with ID starting with d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373 not found: ID does not exist" containerID="d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373"
Oct 29 23:04:09.188426 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.188375    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373"} err="failed to get container status \"d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373\": rpc error: code = NotFound desc = could not find container \"d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373\": container with ID starting with d7b6d0ee0c53344c69cf3c471f514aca79bb9b7bd6b67e289451dd0561962373 not found: ID does not exist"
Oct 29 23:04:09.221993 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.221967    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"]
Oct 29 23:04:09.235251 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.235229    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-22d634-7db4fc9554-56z6p"]
Oct 29 23:04:09.262032 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.261992    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"]
Oct 29 23:04:09.267369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.267348    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-22d634-6768765995-phnhr"]
Oct 29 23:04:09.920617 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:09.920583    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:10.112569 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:10.112535    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35221c46-797c-4c8e-97a5-55d9bc6b4829" path="/var/lib/kubelet/pods/35221c46-797c-4c8e-97a5-55d9bc6b4829/volumes"
Oct 29 23:04:10.113108 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:10.113091    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="601556d3-a9a1-43b8-88b7-320af479646d" path="/var/lib/kubelet/pods/601556d3-a9a1-43b8-88b7-320af479646d/volumes"
Oct 29 23:04:11.634775 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:11.634744    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:11.850159 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:11.850128    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:11.964447 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:11.964412    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:12.119775 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.119741    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"]
Oct 29 23:04:12.120048 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.119967    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l" podUID="50800f81-4679-4607-accf-3b3920cd73d5" containerName="istio-proxy" containerID="cri-o://a2c20822a3d6792a1b9b02f2a1723337fcae410031a6a989a26670911f2f45d3" gracePeriod=30
Oct 29 23:04:12.185094 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.185063    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:12.609424 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.609384    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main" containerID="cri-o://65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c" gracePeriod=45
Oct 29 23:04:12.619960 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.619935    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"]
Oct 29 23:04:12.620406 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620392    4860 cpu_manager.go:414] "containerMap: removing container" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main"
Oct 29 23:04:12.620469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620407    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main"
Oct 29 23:04:12.620469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620423    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f67159e2-0a49-41d4-90d1-fc3626f2c960" containerName="revoke-keys"
Oct 29 23:04:12.620469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620428    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67159e2-0a49-41d4-90d1-fc3626f2c960" containerName="revoke-keys"
Oct 29 23:04:12.620469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620435    4860 cpu_manager.go:414] "containerMap: removing container" podUID="646e1a1e-8f19-4df7-91d8-4864270cae7d" containerName="payload-processing"
Oct 29 23:04:12.620469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620440    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646e1a1e-8f19-4df7-91d8-4864270cae7d" containerName="payload-processing"
Oct 29 23:04:12.620469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620451    4860 cpu_manager.go:414] "containerMap: removing container" podUID="5962c518-2366-4739-995f-0caa8149b9be" containerName="payload-processing"
Oct 29 23:04:12.620469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620456    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5962c518-2366-4739-995f-0caa8149b9be" containerName="payload-processing"
Oct 29 23:04:12.620469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620464    4860 cpu_manager.go:414] "containerMap: removing container" podUID="601556d3-a9a1-43b8-88b7-320af479646d" containerName="payload-pre-processing"
Oct 29 23:04:12.620469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620468    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="601556d3-a9a1-43b8-88b7-320af479646d" containerName="payload-pre-processing"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620481    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620485    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620498    4860 cpu_manager.go:414] "containerMap: removing container" podUID="11e30c8e-fc84-4be0-8a3d-f2e01ac421e6" containerName="revoke-keys"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620503    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e30c8e-fc84-4be0-8a3d-f2e01ac421e6" containerName="revoke-keys"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620515    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" containerName="istio-proxy"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620521    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" containerName="istio-proxy"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620532    4860 cpu_manager.go:414] "containerMap: removing container" podUID="35221c46-797c-4c8e-97a5-55d9bc6b4829" containerName="payload-processing"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620536    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35221c46-797c-4c8e-97a5-55d9bc6b4829" containerName="payload-processing"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620606    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d99f80b-82a0-4f37-b1e2-8d85a6ba1735" containerName="main"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620616    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e30c8e-fc84-4be0-8a3d-f2e01ac421e6" containerName="revoke-keys"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620624    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb9bfa7-8e72-4849-8ef1-b00c24761ba5" containerName="istio-proxy"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620630    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="35221c46-797c-4c8e-97a5-55d9bc6b4829" containerName="payload-processing"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620640    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="482698c2-d44a-4564-b588-d402fd9cac23" containerName="main"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620646    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67159e2-0a49-41d4-90d1-fc3626f2c960" containerName="revoke-keys"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620651    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="646e1a1e-8f19-4df7-91d8-4864270cae7d" containerName="payload-processing"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620656    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5962c518-2366-4739-995f-0caa8149b9be" containerName="payload-processing"
Oct 29 23:04:12.620753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.620661    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="601556d3-a9a1-43b8-88b7-320af479646d" containerName="payload-pre-processing"
Oct 29 23:04:12.623952 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.623936    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:12.642450 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.642426    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"]
Oct 29 23:04:12.651937 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.651909    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:04:12.652031 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.651971    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:04:12.745887 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.745854    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqrv5\" (UniqueName: \"kubernetes.io/projected/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-kube-api-access-jqrv5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8\" (UID: \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:12.746027 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.745895    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8\" (UID: \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:12.746027 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.746003    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8\" (UID: \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:12.846798 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.846765    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8\" (UID: \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:12.846990 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.846967    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrv5\" (UniqueName: \"kubernetes.io/projected/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-kube-api-access-jqrv5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8\" (UID: \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:12.847047 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.847020    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8\" (UID: \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:12.847849 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.847823    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8\" (UID: \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:12.849792 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.849774    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8\" (UID: \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:12.867150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.867097    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqrv5\" (UniqueName: \"kubernetes.io/projected/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-kube-api-access-jqrv5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8\" (UID: \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:12.887168 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.887147    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 23:04:12.947622 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.947595    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-dshm\") pod \"5837adb2-8517-4f79-97c0-81535c4a9f8a\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") "
Oct 29 23:04:12.947765 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.947687    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5837adb2-8517-4f79-97c0-81535c4a9f8a-kube-api-access-2pxxq\" (UniqueName: \"kubernetes.io/projected/5837adb2-8517-4f79-97c0-81535c4a9f8a-kube-api-access-2pxxq\") pod \"5837adb2-8517-4f79-97c0-81535c4a9f8a\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") "
Oct 29 23:04:12.947847 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.947828    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5837adb2-8517-4f79-97c0-81535c4a9f8a-tls-certs\" (UniqueName: \"kubernetes.io/secret/5837adb2-8517-4f79-97c0-81535c4a9f8a-tls-certs\") pod \"5837adb2-8517-4f79-97c0-81535c4a9f8a\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") "
Oct 29 23:04:12.947921 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.947904    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-home\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-home\") pod \"5837adb2-8517-4f79-97c0-81535c4a9f8a\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") "
Oct 29 23:04:12.947989 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.947967    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-model-cache\") pod \"5837adb2-8517-4f79-97c0-81535c4a9f8a\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") "
Oct 29 23:04:12.948058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.948021    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-tmp-dir\") pod \"5837adb2-8517-4f79-97c0-81535c4a9f8a\" (UID: \"5837adb2-8517-4f79-97c0-81535c4a9f8a\") "
Oct 29 23:04:12.948211 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.948187    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-home" pod "5837adb2-8517-4f79-97c0-81535c4a9f8a" (UID: "5837adb2-8517-4f79-97c0-81535c4a9f8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:12.948303 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.948278    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-model-cache" pod "5837adb2-8517-4f79-97c0-81535c4a9f8a" (UID: "5837adb2-8517-4f79-97c0-81535c4a9f8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:12.948406 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.948387    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-tmp-dir" pod "5837adb2-8517-4f79-97c0-81535c4a9f8a" (UID: "5837adb2-8517-4f79-97c0-81535c4a9f8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:12.948583 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.948566    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:12.948650 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.948586    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:12.948650 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.948606    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:12.950096 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.950062    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5837adb2-8517-4f79-97c0-81535c4a9f8a-kube-api-access-2pxxq" pod "5837adb2-8517-4f79-97c0-81535c4a9f8a" (UID: "5837adb2-8517-4f79-97c0-81535c4a9f8a"). InnerVolumeSpecName "kube-api-access-2pxxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:12.950719 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.950698    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-dshm" pod "5837adb2-8517-4f79-97c0-81535c4a9f8a" (UID: "5837adb2-8517-4f79-97c0-81535c4a9f8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:12.950778 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.950701    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5837adb2-8517-4f79-97c0-81535c4a9f8a-tls-certs" pod "5837adb2-8517-4f79-97c0-81535c4a9f8a" (UID: "5837adb2-8517-4f79-97c0-81535c4a9f8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:12.988925 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:12.988900    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:13.050247 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.050211    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5837adb2-8517-4f79-97c0-81535c4a9f8a-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:13.050247 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.050242    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5837adb2-8517-4f79-97c0-81535c4a9f8a-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:13.050490 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.050255    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pxxq\" (UniqueName: \"kubernetes.io/projected/5837adb2-8517-4f79-97c0-81535c4a9f8a-kube-api-access-2pxxq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:13.209837 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.209799    4860 generic.go:356] "Generic (PLEG): container finished" podID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerID="65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c" exitCode=0
Oct 29 23:04:13.210016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.209860    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" event={"ID":"5837adb2-8517-4f79-97c0-81535c4a9f8a","Type":"ContainerDied","Data":"65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c"}
Oct 29 23:04:13.210016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.209891    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n" event={"ID":"5837adb2-8517-4f79-97c0-81535c4a9f8a","Type":"ContainerDied","Data":"873025e2e93ffed3ca3b5a8d6e8a10d927fb90bf638c57251197c56f7e3b094e"}
Oct 29 23:04:13.210016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.209907    4860 scope.go:122] "RemoveContainer" containerID="65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c"
Oct 29 23:04:13.210016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.209863    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"
Oct 29 23:04:13.220573 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.220553    4860 scope.go:122] "RemoveContainer" containerID="65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c"
Oct 29 23:04:13.220814 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:04:13.220796    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c\": container with ID starting with 65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c not found: ID does not exist" containerID="65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c"
Oct 29 23:04:13.220904 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.220820    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c"} err="failed to get container status \"65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c\": rpc error: code = NotFound desc = could not find container \"65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c\": container with ID starting with 65ac413f5f3dde63fdfc1379b69d683d4d1b1ce0e6e7ffa037a8c679b9a9068c not found: ID does not exist"
Oct 29 23:04:13.224689 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.224669    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:13.267678 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.267639    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"]
Oct 29 23:04:13.278347 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.278295    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-22d634/simulator-w5-22d634-kserve-84ff4d869f-nx68n"]
Oct 29 23:04:13.304608 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:04:13.304571    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c21cd0_fd07_47f5_bd77_91cdb5155ac8.slice/crio-c5f5280ba2e84ec876b61ad8fe6edb051c347b8a639a0704f835c75b557f8997 WatchSource:0}: Error finding container c5f5280ba2e84ec876b61ad8fe6edb051c347b8a639a0704f835c75b557f8997: Status 404 returned error can't find the container with id c5f5280ba2e84ec876b61ad8fe6edb051c347b8a639a0704f835c75b557f8997
Oct 29 23:04:13.305722 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:13.305701    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"]
Oct 29 23:04:14.116444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:14.116412    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" path="/var/lib/kubelet/pods/5837adb2-8517-4f79-97c0-81535c4a9f8a/volumes"
Oct 29 23:04:14.219118 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:14.219090    4860 generic.go:356] "Generic (PLEG): container finished" podID="23c21cd0-fd07-47f5-bd77-91cdb5155ac8" containerID="8cc1c883a0833f5110ba41d84aefadd1e2d4ff84e6b55dd53723e3a4c8867567" exitCode=0
Oct 29 23:04:14.219284 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:14.219175    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8" event={"ID":"23c21cd0-fd07-47f5-bd77-91cdb5155ac8","Type":"ContainerDied","Data":"8cc1c883a0833f5110ba41d84aefadd1e2d4ff84e6b55dd53723e3a4c8867567"}
Oct 29 23:04:14.219284 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:14.219227    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8" event={"ID":"23c21cd0-fd07-47f5-bd77-91cdb5155ac8","Type":"ContainerStarted","Data":"c5f5280ba2e84ec876b61ad8fe6edb051c347b8a639a0704f835c75b557f8997"}
Oct 29 23:04:15.044717 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:15.044684    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:15.377536 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:15.377512    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:15.482375 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:15.482346    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-kube-api-access-jqrv5\" (UniqueName: \"kubernetes.io/projected/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-kube-api-access-jqrv5\") pod \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\" (UID: \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\") "
Oct 29 23:04:15.482540 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:15.482388    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-openshift-ca\") pod \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\" (UID: \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\") "
Oct 29 23:04:15.482540 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:15.482444    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-xks-ca\") pod \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\" (UID: \"23c21cd0-fd07-47f5-bd77-91cdb5155ac8\") "
Oct 29 23:04:15.482803 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:15.482777    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-openshift-ca" pod "23c21cd0-fd07-47f5-bd77-91cdb5155ac8" (UID: "23c21cd0-fd07-47f5-bd77-91cdb5155ac8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:15.484463 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:15.484439    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-xks-ca" pod "23c21cd0-fd07-47f5-bd77-91cdb5155ac8" (UID: "23c21cd0-fd07-47f5-bd77-91cdb5155ac8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:15.484538 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:15.484486    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-kube-api-access-jqrv5" pod "23c21cd0-fd07-47f5-bd77-91cdb5155ac8" (UID: "23c21cd0-fd07-47f5-bd77-91cdb5155ac8"). InnerVolumeSpecName "kube-api-access-jqrv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:15.583530 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:15.583507    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqrv5\" (UniqueName: \"kubernetes.io/projected/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-kube-api-access-jqrv5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:15.583530 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:15.583530    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:15.583666 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:15.583541    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c21cd0-fd07-47f5-bd77-91cdb5155ac8-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:16.234222 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:16.234192    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"
Oct 29 23:04:16.234408 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:16.234231    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8" event={"ID":"23c21cd0-fd07-47f5-bd77-91cdb5155ac8","Type":"ContainerDied","Data":"c5f5280ba2e84ec876b61ad8fe6edb051c347b8a639a0704f835c75b557f8997"}
Oct 29 23:04:16.234408 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:16.234275    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5f5280ba2e84ec876b61ad8fe6edb051c347b8a639a0704f835c75b557f8997"
Oct 29 23:04:16.453388 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:16.453341    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:16.810622 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:16.810550    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:17.243020 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.242993    4860 generic.go:356] "Generic (PLEG): container finished" podID="50800f81-4679-4607-accf-3b3920cd73d5" containerID="a2c20822a3d6792a1b9b02f2a1723337fcae410031a6a989a26670911f2f45d3" exitCode=0
Oct 29 23:04:17.243132 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.243066    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l" event={"ID":"50800f81-4679-4607-accf-3b3920cd73d5","Type":"ContainerDied","Data":"a2c20822a3d6792a1b9b02f2a1723337fcae410031a6a989a26670911f2f45d3"}
Oct 29 23:04:17.392425 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.392369    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:04:17.504046 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504016    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50800f81-4679-4607-accf-3b3920cd73d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50800f81-4679-4607-accf-3b3920cd73d5-istio-podinfo\") pod \"50800f81-4679-4607-accf-3b3920cd73d5\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") "
Oct 29 23:04:17.504486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504066    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-envoy\") pod \"50800f81-4679-4607-accf-3b3920cd73d5\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") "
Oct 29 23:04:17.504486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504111    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50800f81-4679-4607-accf-3b3920cd73d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50800f81-4679-4607-accf-3b3920cd73d5-istiod-ca-cert\") pod \"50800f81-4679-4607-accf-3b3920cd73d5\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") "
Oct 29 23:04:17.504486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504148    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-data\") pod \"50800f81-4679-4607-accf-3b3920cd73d5\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") "
Oct 29 23:04:17.504486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504175    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-certs\") pod \"50800f81-4679-4607-accf-3b3920cd73d5\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") "
Oct 29 23:04:17.504486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504192    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-socket\") pod \"50800f81-4679-4607-accf-3b3920cd73d5\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") "
Oct 29 23:04:17.504486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504224    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-kube-api-access-lkbgj\" (UniqueName: \"kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-kube-api-access-lkbgj\") pod \"50800f81-4679-4607-accf-3b3920cd73d5\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") "
Oct 29 23:04:17.504486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504295    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-credential-socket\") pod \"50800f81-4679-4607-accf-3b3920cd73d5\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") "
Oct 29 23:04:17.504486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504449    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-istio-token\" (UniqueName: \"kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-istio-token\") pod \"50800f81-4679-4607-accf-3b3920cd73d5\" (UID: \"50800f81-4679-4607-accf-3b3920cd73d5\") "
Oct 29 23:04:17.504987 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504582    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-socket" pod "50800f81-4679-4607-accf-3b3920cd73d5" (UID: "50800f81-4679-4607-accf-3b3920cd73d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.504987 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504636    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50800f81-4679-4607-accf-3b3920cd73d5-istiod-ca-cert" pod "50800f81-4679-4607-accf-3b3920cd73d5" (UID: "50800f81-4679-4607-accf-3b3920cd73d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:17.504987 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504595    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-certs" pod "50800f81-4679-4607-accf-3b3920cd73d5" (UID: "50800f81-4679-4607-accf-3b3920cd73d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.504987 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504810    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-credential-socket" pod "50800f81-4679-4607-accf-3b3920cd73d5" (UID: "50800f81-4679-4607-accf-3b3920cd73d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.504987 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.504912    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-data" pod "50800f81-4679-4607-accf-3b3920cd73d5" (UID: "50800f81-4679-4607-accf-3b3920cd73d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.505151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.505068    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50800f81-4679-4607-accf-3b3920cd73d5-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.505151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.505088    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.505151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.505101    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.505151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.505114    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.505151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.505127    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.506400 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.506374    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50800f81-4679-4607-accf-3b3920cd73d5-istio-podinfo" pod "50800f81-4679-4607-accf-3b3920cd73d5" (UID: "50800f81-4679-4607-accf-3b3920cd73d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:17.506741 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.506722    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-istio-token" pod "50800f81-4679-4607-accf-3b3920cd73d5" (UID: "50800f81-4679-4607-accf-3b3920cd73d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:17.506914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.506900    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-envoy" pod "50800f81-4679-4607-accf-3b3920cd73d5" (UID: "50800f81-4679-4607-accf-3b3920cd73d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:17.506982 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.506955    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-kube-api-access-lkbgj" pod "50800f81-4679-4607-accf-3b3920cd73d5" (UID: "50800f81-4679-4607-accf-3b3920cd73d5"). InnerVolumeSpecName "kube-api-access-lkbgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:17.606414 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.606388    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50800f81-4679-4607-accf-3b3920cd73d5-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.606414 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.606411    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50800f81-4679-4607-accf-3b3920cd73d5-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.606561 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.606420    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkbgj\" (UniqueName: \"kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-kube-api-access-lkbgj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:17.606561 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:17.606430    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50800f81-4679-4607-accf-3b3920cd73d5-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:18.058218 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:04:18.058168    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:04:18.251643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:18.251613    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"
Oct 29 23:04:18.251643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:18.251637    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l" event={"ID":"50800f81-4679-4607-accf-3b3920cd73d5","Type":"ContainerDied","Data":"6d28f6dcbc473a10db560818a58212ad2101aac353cfb9d15d4aa2e86c5ff363"}
Oct 29 23:04:18.251896 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:18.251695    4860 scope.go:122] "RemoveContainer" containerID="a2c20822a3d6792a1b9b02f2a1723337fcae410031a6a989a26670911f2f45d3"
Oct 29 23:04:18.321850 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:18.321789    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"]
Oct 29 23:04:18.325843 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:18.325817    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d6d4e-openshift-default-5b7b554885-5j85l"]
Oct 29 23:04:18.336800 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:18.336776    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:18.792484 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:18.792460    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 23:04:18.820358 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:18.820312    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 23:04:19.815823 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:19.815788    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:20.111609 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:20.111530    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50800f81-4679-4607-accf-3b3920cd73d5" path="/var/lib/kubelet/pods/50800f81-4679-4607-accf-3b3920cd73d5/volumes"
Oct 29 23:04:21.362382 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:21.362350    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:22.894439 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:22.894404    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:23.233643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:23.233608    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:23.356543 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:23.356503    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:24.550572 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:24.550532    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:25.013757 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:25.013720    4860 scope.go:122] "RemoveContainer" containerID="a16dd42fdd30462f4d8fe0153772104d4a28d270085105fb5b281cebfa2d5f19"
Oct 29 23:04:25.023999 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:25.023980    4860 scope.go:122] "RemoveContainer" containerID="bd136255f162d32b41eb38f1d180b3900d5f12ecccdeb73345e43bb23d989f51"
Oct 29 23:04:25.033227 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:25.033209    4860 scope.go:122] "RemoveContainer" containerID="d0a3d9f177844dd68348fef059cc99456cc33ea00643cbb339d2e86df604d341"
Oct 29 23:04:25.042115 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:25.042096    4860 scope.go:122] "RemoveContainer" containerID="2b3e370ff1a0b98ede9c4715cf0f5c96c7313acf5198f226782c933bc54c5283"
Oct 29 23:04:25.050741 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:25.050722    4860 scope.go:122] "RemoveContainer" containerID="c249a1965397b2d33bea46f2a3ca5fd136f4e151d7cfe7831802b5dd8175003d"
Oct 29 23:04:25.062484 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:25.062464    4860 scope.go:122] "RemoveContainer" containerID="2edc608078c6c34526400eeec3a1ae5ba4da5b0f7032aeee49411373a71e3340"
Oct 29 23:04:25.071178 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:25.071161    4860 scope.go:122] "RemoveContainer" containerID="bd520a3cffa2126811b554290c65a09bb2823d9c440de7c69bec4cc8db1a60c4"
Oct 29 23:04:25.079367 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:25.079306    4860 scope.go:122] "RemoveContainer" containerID="a425528dba76db0abf9c20e1989688d9bcdba98e072d5186844c1c235a2bb1cf"
Oct 29 23:04:25.823124 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:25.823091    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:26.284394 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:26.284362    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:26.407083 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:26.407048    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"]
Oct 29 23:04:26.408692 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:26.408659    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:26.419800 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:26.419766    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-af260f-h74d8"]
Oct 29 23:04:26.518804 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:26.518765    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:26.897224 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:26.897193    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:28.112281 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:28.112250    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c21cd0-fd07-47f5-bd77-91cdb5155ac8" path="/var/lib/kubelet/pods/23c21cd0-fd07-47f5-bd77-91cdb5155ac8/volumes"
Oct 29 23:04:28.144504 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:28.144476    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:29.486477 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:29.486429    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:29.944487 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:29.944448    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:30.072823 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:30.072789    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:30.577286 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:30.577245    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:31.824254 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:31.824218    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:33.137164 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:33.137125    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:33.253910 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:33.253877    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:33.383206 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:33.382581    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:33.403186 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:33.403104    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"]
Oct 29 23:04:33.403415 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:33.403381    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt" podUID="ecb268da-f8b2-4be8-b379-f96e4eb818fc" containerName="istio-proxy" containerID="cri-o://751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d" gracePeriod=30
Oct 29 23:04:33.414440 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:33.414270    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"]
Oct 29 23:04:33.415128 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:33.414497    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy" containerID="cri-o://cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2" gracePeriod=30
Oct 29 23:04:33.528407 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:33.528365    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:35.038588 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:35.038538    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:35.410618 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:35.410512    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:35.614288 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:35.614245    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:35.737272 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:35.737243    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:36.754313 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:36.754281    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:37.064652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:37.064572    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:37.194885 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:37.194854    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:37.367686 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:37.367603    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"]
Oct 29 23:04:37.367862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:37.367808    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn" podUID="d13f5a1b-9637-4492-81c3-874e6c91294d" containerName="istio-proxy" containerID="cri-o://0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83" gracePeriod=30
Oct 29 23:04:37.435651 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:37.435616    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:38.416223 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.416184    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:38.678821 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.678765    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 23:04:38.710452 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.710432    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 23:04:38.733631 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.733607    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-data\") pod \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") "
Oct 29 23:04:38.733742 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.733691    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-token\" (UniqueName: \"kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-token\") pod \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") "
Oct 29 23:04:38.733742 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.733722    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-certs\") pod \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") "
Oct 29 23:04:38.733855 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.733740    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-podinfo\") pod \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") "
Oct 29 23:04:38.733855 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.733769    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-envoy\") pod \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") "
Oct 29 23:04:38.733855 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.733820    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istiod-ca-cert\") pod \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") "
Oct 29 23:04:38.733855 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.733848    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-socket\") pod \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") "
Oct 29 23:04:38.734029 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.733892    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-data" pod "ecb268da-f8b2-4be8-b379-f96e4eb818fc" (UID: "ecb268da-f8b2-4be8-b379-f96e4eb818fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.734029 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.733897    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-kube-api-access-cs8wm\" (UniqueName: \"kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-kube-api-access-cs8wm\") pod \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") "
Oct 29 23:04:38.734029 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.734005    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-credential-socket\") pod \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\" (UID: \"ecb268da-f8b2-4be8-b379-f96e4eb818fc\") "
Oct 29 23:04:38.734300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.734277    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-certs" pod "ecb268da-f8b2-4be8-b379-f96e4eb818fc" (UID: "ecb268da-f8b2-4be8-b379-f96e4eb818fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.734760 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.734606    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.734760 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.734631    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.734760 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.734647    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-credential-socket" pod "ecb268da-f8b2-4be8-b379-f96e4eb818fc" (UID: "ecb268da-f8b2-4be8-b379-f96e4eb818fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.734760 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.734719    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istiod-ca-cert" pod "ecb268da-f8b2-4be8-b379-f96e4eb818fc" (UID: "ecb268da-f8b2-4be8-b379-f96e4eb818fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:38.735415 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.735387    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-socket" pod "ecb268da-f8b2-4be8-b379-f96e4eb818fc" (UID: "ecb268da-f8b2-4be8-b379-f96e4eb818fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.737225 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.737186    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-envoy" pod "ecb268da-f8b2-4be8-b379-f96e4eb818fc" (UID: "ecb268da-f8b2-4be8-b379-f96e4eb818fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.737806 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.737782    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-kube-api-access-cs8wm" pod "ecb268da-f8b2-4be8-b379-f96e4eb818fc" (UID: "ecb268da-f8b2-4be8-b379-f96e4eb818fc"). InnerVolumeSpecName "kube-api-access-cs8wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:38.738141 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.738120    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-podinfo" pod "ecb268da-f8b2-4be8-b379-f96e4eb818fc" (UID: "ecb268da-f8b2-4be8-b379-f96e4eb818fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:38.742469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.742440    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-token" pod "ecb268da-f8b2-4be8-b379-f96e4eb818fc" (UID: "ecb268da-f8b2-4be8-b379-f96e4eb818fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:38.835299 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835271    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-podinfo\") pod \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") "
Oct 29 23:04:38.835451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835361    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-data\") pod \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") "
Oct 29 23:04:38.835451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835379    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-credential-socket\") pod \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") "
Oct 29 23:04:38.835451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835399    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-socket\") pod \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") "
Oct 29 23:04:38.835451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835431    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istiod-ca-cert\") pod \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") "
Oct 29 23:04:38.835666 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835483    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-envoy\") pod \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") "
Oct 29 23:04:38.835666 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835591    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-token\" (UniqueName: \"kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-token\") pod \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") "
Oct 29 23:04:38.835666 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835633    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-certs\") pod \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") "
Oct 29 23:04:38.835829 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835695    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-kube-api-access-pck6b\" (UniqueName: \"kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-kube-api-access-pck6b\") pod \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\" (UID: \"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0\") "
Oct 29 23:04:38.835829 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835708    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-socket" pod "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" (UID: "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.835829 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835722    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-data" pod "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" (UID: "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.835829 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835739    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-credential-socket" pod "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" (UID: "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.836069 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835947    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-certs" pod "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" (UID: "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.836069 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.835982    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istiod-ca-cert" pod "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" (UID: "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:38.836249 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.836230    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.836319 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.836255    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.836319 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.836294    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.836319 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.836308    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.836319 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.836337    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.836549 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.836353    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs8wm\" (UniqueName: \"kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-kube-api-access-cs8wm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.836549 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.836367    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb268da-f8b2-4be8-b379-f96e4eb818fc-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.836549 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.836380    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.836549 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.836394    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.836549 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.836408    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.836549 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.836421    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.836549 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.836433    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb268da-f8b2-4be8-b379-f96e4eb818fc-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.837429 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.837401    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-podinfo" pod "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" (UID: "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:38.837752 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.837733    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-token" pod "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" (UID: "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:38.838208 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.838187    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-envoy" pod "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" (UID: "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:38.838283 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.838203    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-kube-api-access-pck6b" pod "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" (UID: "a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0"). InnerVolumeSpecName "kube-api-access-pck6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:38.937718 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.937698    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.937718 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.937719    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pck6b\" (UniqueName: \"kubernetes.io/projected/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-kube-api-access-pck6b\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.937848 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.937728    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:38.937848 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:38.937738    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:39.403688 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.403608    4860 generic.go:356] "Generic (PLEG): container finished" podID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerID="cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2" exitCode=0
Oct 29 23:04:39.403688 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.403659    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"
Oct 29 23:04:39.403895 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.403692    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" event={"ID":"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0","Type":"ContainerDied","Data":"cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2"}
Oct 29 23:04:39.403895 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.403747    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw" event={"ID":"a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0","Type":"ContainerDied","Data":"630ecf9cb75353c6c6d4f49693b18d6d4c5f3480bc0e41787ab3392464f1d775"}
Oct 29 23:04:39.403895 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.403775    4860 scope.go:122] "RemoveContainer" containerID="cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2"
Oct 29 23:04:39.405291 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.405265    4860 generic.go:356] "Generic (PLEG): container finished" podID="ecb268da-f8b2-4be8-b379-f96e4eb818fc" containerID="751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d" exitCode=0
Oct 29 23:04:39.405490 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.405337    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"
Oct 29 23:04:39.405490 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.405352    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt" event={"ID":"ecb268da-f8b2-4be8-b379-f96e4eb818fc","Type":"ContainerDied","Data":"751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d"}
Oct 29 23:04:39.405490 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.405404    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt" event={"ID":"ecb268da-f8b2-4be8-b379-f96e4eb818fc","Type":"ContainerDied","Data":"7f8add58ad8aee947586a07b8db4cc13ef80b84058f57e0e8926360ecd5c0c4d"}
Oct 29 23:04:39.414065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.414017    4860 scope.go:122] "RemoveContainer" containerID="e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70"
Oct 29 23:04:39.423908 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.423891    4860 scope.go:122] "RemoveContainer" containerID="cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2"
Oct 29 23:04:39.424209 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:04:39.424140    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2\": container with ID starting with cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2 not found: ID does not exist" containerID="cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2"
Oct 29 23:04:39.424209 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.424165    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2"} err="failed to get container status \"cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2\": rpc error: code = NotFound desc = could not find container \"cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2\": container with ID starting with cc11e16010586bc732cc78ec8daee4834723b20cdaa7f508d187763ee111d5c2 not found: ID does not exist"
Oct 29 23:04:39.424209 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.424181    4860 scope.go:122] "RemoveContainer" containerID="e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70"
Oct 29 23:04:39.424516 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:04:39.424470    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70\": container with ID starting with e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70 not found: ID does not exist" containerID="e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70"
Oct 29 23:04:39.424516 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.424493    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70"} err="failed to get container status \"e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70\": rpc error: code = NotFound desc = could not find container \"e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70\": container with ID starting with e3df6dfad6cc841d0d6b61df694d763b2e44d962cecc72fde298914b1fb4bb70 not found: ID does not exist"
Oct 29 23:04:39.424516 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.424508    4860 scope.go:122] "RemoveContainer" containerID="751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d"
Oct 29 23:04:39.443617 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.443591    4860 scope.go:122] "RemoveContainer" containerID="751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d"
Oct 29 23:04:39.443966 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:04:39.443900    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d\": container with ID starting with 751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d not found: ID does not exist" containerID="751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d"
Oct 29 23:04:39.443966 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.443924    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d"} err="failed to get container status \"751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d\": rpc error: code = NotFound desc = could not find container \"751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d\": container with ID starting with 751d9cd712cd5088c29f391bb6bf6844d426f7209667cd3bbd7c6cfa8d83306d not found: ID does not exist"
Oct 29 23:04:39.459602 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.459580    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"]
Oct 29 23:04:39.472391 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.472371    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-cxjmw"]
Oct 29 23:04:39.497090 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.497066    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"]
Oct 29 23:04:39.500251 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.500232    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-af260f-openshift-default-846bb69f67-ftzdt"]
Oct 29 23:04:39.990126 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:39.990089    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:40.112035 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:40.112003    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" path="/var/lib/kubelet/pods/a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0/volumes"
Oct 29 23:04:40.112561 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:40.112545    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb268da-f8b2-4be8-b379-f96e4eb818fc" path="/var/lib/kubelet/pods/ecb268da-f8b2-4be8-b379-f96e4eb818fc/volumes"
Oct 29 23:04:41.403039 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:41.403005    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:41.576489 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:41.576454    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:42.624283 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.624260    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 23:04:42.674989 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.674932    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-kube-api-access-cnm8c\" (UniqueName: \"kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-kube-api-access-cnm8c\") pod \"d13f5a1b-9637-4492-81c3-874e6c91294d\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") "
Oct 29 23:04:42.674989 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.674967    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13f5a1b-9637-4492-81c3-874e6c91294d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13f5a1b-9637-4492-81c3-874e6c91294d-istiod-ca-cert\") pod \"d13f5a1b-9637-4492-81c3-874e6c91294d\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") "
Oct 29 23:04:42.675135 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.674990    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-podinfo\") pod \"d13f5a1b-9637-4492-81c3-874e6c91294d\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") "
Oct 29 23:04:42.675135 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.675078    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-envoy\") pod \"d13f5a1b-9637-4492-81c3-874e6c91294d\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") "
Oct 29 23:04:42.675135 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.675097    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-token\" (UniqueName: \"kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-token\") pod \"d13f5a1b-9637-4492-81c3-874e6c91294d\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") "
Oct 29 23:04:42.675135 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.675132    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-data\") pod \"d13f5a1b-9637-4492-81c3-874e6c91294d\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") "
Oct 29 23:04:42.675444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.675233    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-credential-socket\") pod \"d13f5a1b-9637-4492-81c3-874e6c91294d\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") "
Oct 29 23:04:42.675444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.675305    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-socket\") pod \"d13f5a1b-9637-4492-81c3-874e6c91294d\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") "
Oct 29 23:04:42.675444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.675429    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-certs\") pod \"d13f5a1b-9637-4492-81c3-874e6c91294d\" (UID: \"d13f5a1b-9637-4492-81c3-874e6c91294d\") "
Oct 29 23:04:42.675609 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.675485    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-data" pod "d13f5a1b-9637-4492-81c3-874e6c91294d" (UID: "d13f5a1b-9637-4492-81c3-874e6c91294d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:42.675769 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.675735    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-credential-socket" pod "d13f5a1b-9637-4492-81c3-874e6c91294d" (UID: "d13f5a1b-9637-4492-81c3-874e6c91294d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:42.676004 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.675982    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.676105 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.676016    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-socket" pod "d13f5a1b-9637-4492-81c3-874e6c91294d" (UID: "d13f5a1b-9637-4492-81c3-874e6c91294d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:42.676105 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.676029    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.676577 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.676541    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-certs" pod "d13f5a1b-9637-4492-81c3-874e6c91294d" (UID: "d13f5a1b-9637-4492-81c3-874e6c91294d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:42.676734 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.676711    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13f5a1b-9637-4492-81c3-874e6c91294d-istiod-ca-cert" pod "d13f5a1b-9637-4492-81c3-874e6c91294d" (UID: "d13f5a1b-9637-4492-81c3-874e6c91294d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:42.677836 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.677811    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-envoy" pod "d13f5a1b-9637-4492-81c3-874e6c91294d" (UID: "d13f5a1b-9637-4492-81c3-874e6c91294d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:04:42.678561 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.678536    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-kube-api-access-cnm8c" pod "d13f5a1b-9637-4492-81c3-874e6c91294d" (UID: "d13f5a1b-9637-4492-81c3-874e6c91294d"). InnerVolumeSpecName "kube-api-access-cnm8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:42.678765 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.678734    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-podinfo" pod "d13f5a1b-9637-4492-81c3-874e6c91294d" (UID: "d13f5a1b-9637-4492-81c3-874e6c91294d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:04:42.678867 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.678834    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-token" pod "d13f5a1b-9637-4492-81c3-874e6c91294d" (UID: "d13f5a1b-9637-4492-81c3-874e6c91294d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:42.777515 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.777494    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.777515 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.777515    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.777659 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.777525    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.777659 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.777533    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13f5a1b-9637-4492-81c3-874e6c91294d-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.777659 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.777544    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnm8c\" (UniqueName: \"kubernetes.io/projected/d13f5a1b-9637-4492-81c3-874e6c91294d-kube-api-access-cnm8c\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.777659 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.777552    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13f5a1b-9637-4492-81c3-874e6c91294d-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.777659 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:42.777560    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13f5a1b-9637-4492-81c3-874e6c91294d-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:43.258506 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:43.258475    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:43.435687 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:43.435657    4860 generic.go:356] "Generic (PLEG): container finished" podID="d13f5a1b-9637-4492-81c3-874e6c91294d" containerID="0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83" exitCode=0
Oct 29 23:04:43.435875 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:43.435705    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"
Oct 29 23:04:43.435875 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:43.435741    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn" event={"ID":"d13f5a1b-9637-4492-81c3-874e6c91294d","Type":"ContainerDied","Data":"0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83"}
Oct 29 23:04:43.435875 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:43.435792    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn" event={"ID":"d13f5a1b-9637-4492-81c3-874e6c91294d","Type":"ContainerDied","Data":"4cf8902949ad318eace7d10f03cdc4b19222e2a49c37e1d7a5ce071bc997fd3b"}
Oct 29 23:04:43.435875 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:43.435814    4860 scope.go:122] "RemoveContainer" containerID="0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83"
Oct 29 23:04:43.446451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:43.446431    4860 scope.go:122] "RemoveContainer" containerID="0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83"
Oct 29 23:04:43.446688 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:04:43.446670    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83\": container with ID starting with 0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83 not found: ID does not exist" containerID="0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83"
Oct 29 23:04:43.446739 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:43.446699    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83"} err="failed to get container status \"0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83\": rpc error: code = NotFound desc = could not find container \"0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83\": container with ID starting with 0d35f4fb70a11e93165ed064c480592b6b4581ac4341b40da11da5f484cbca83 not found: ID does not exist"
Oct 29 23:04:43.506307 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:43.506286    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"]
Oct 29 23:04:43.519160 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:43.519106    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-22d634-openshift-default-89dbbf78f-4jtkn"]
Oct 29 23:04:44.113072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:44.113038    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13f5a1b-9637-4492-81c3-874e6c91294d" path="/var/lib/kubelet/pods/d13f5a1b-9637-4492-81c3-874e6c91294d/volumes"
Oct 29 23:04:44.492561 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:44.492523    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:44.595101 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:44.595065    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:44.733055 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:44.733018    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:46.253074 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:46.253042    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:47.439776 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.439740    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"]
Oct 29 23:04:47.440256 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.439961    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5" podUID="b2604c6c-436e-4aa2-9ec6-fe306749909d" containerName="maas-api" containerID="cri-o://7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7" gracePeriod=30
Oct 29 23:04:47.519624 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.519594    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:47.711559 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.711531    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 23:04:47.827786 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.827756    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-metrics-certs\") pod \"b2604c6c-436e-4aa2-9ec6-fe306749909d\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") "
Oct 29 23:04:47.827786 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.827790    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-maas-api-tls\") pod \"b2604c6c-436e-4aa2-9ec6-fe306749909d\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") "
Oct 29 23:04:47.828017 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.827817    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2604c6c-436e-4aa2-9ec6-fe306749909d-kube-api-access-xr7k8\" (UniqueName: \"kubernetes.io/projected/b2604c6c-436e-4aa2-9ec6-fe306749909d-kube-api-access-xr7k8\") pod \"b2604c6c-436e-4aa2-9ec6-fe306749909d\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") "
Oct 29 23:04:47.828017 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.827970    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2604c6c-436e-4aa2-9ec6-fe306749909d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2604c6c-436e-4aa2-9ec6-fe306749909d-openshift-service-ca\") pod \"b2604c6c-436e-4aa2-9ec6-fe306749909d\" (UID: \"b2604c6c-436e-4aa2-9ec6-fe306749909d\") "
Oct 29 23:04:47.828415 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.828387    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2604c6c-436e-4aa2-9ec6-fe306749909d-openshift-service-ca" pod "b2604c6c-436e-4aa2-9ec6-fe306749909d" (UID: "b2604c6c-436e-4aa2-9ec6-fe306749909d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:47.830025 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.829999    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-metrics-certs" pod "b2604c6c-436e-4aa2-9ec6-fe306749909d" (UID: "b2604c6c-436e-4aa2-9ec6-fe306749909d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:47.830025 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.830020    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2604c6c-436e-4aa2-9ec6-fe306749909d-kube-api-access-xr7k8" pod "b2604c6c-436e-4aa2-9ec6-fe306749909d" (UID: "b2604c6c-436e-4aa2-9ec6-fe306749909d"). InnerVolumeSpecName "kube-api-access-xr7k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:47.830177 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.830016    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-maas-api-tls" pod "b2604c6c-436e-4aa2-9ec6-fe306749909d" (UID: "b2604c6c-436e-4aa2-9ec6-fe306749909d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:47.929252 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.929217    4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-metrics-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:47.929252 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.929246    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2604c6c-436e-4aa2-9ec6-fe306749909d-maas-api-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:47.929252 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.929258    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr7k8\" (UniqueName: \"kubernetes.io/projected/b2604c6c-436e-4aa2-9ec6-fe306749909d-kube-api-access-xr7k8\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:47.929547 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.929267    4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2604c6c-436e-4aa2-9ec6-fe306749909d-openshift-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:04:47.954577 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:47.954512    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:48.057465 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:04:48.057427    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:04:48.071345 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:48.071298    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:48.477136 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:48.477104    4860 generic.go:356] "Generic (PLEG): container finished" podID="b2604c6c-436e-4aa2-9ec6-fe306749909d" containerID="7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7" exitCode=0
Oct 29 23:04:48.477594 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:48.477145    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"
Oct 29 23:04:48.477594 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:48.477152    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5" event={"ID":"b2604c6c-436e-4aa2-9ec6-fe306749909d","Type":"ContainerDied","Data":"7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7"}
Oct 29 23:04:48.477594 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:48.477203    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5" event={"ID":"b2604c6c-436e-4aa2-9ec6-fe306749909d","Type":"ContainerDied","Data":"c5c1c1f67d795b997ffc932943fd9c80e00ff2eae45dfc3e4f3101f9798bc69d"}
Oct 29 23:04:48.477594 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:48.477236    4860 scope.go:122] "RemoveContainer" containerID="7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7"
Oct 29 23:04:48.490116 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:48.490099    4860 scope.go:122] "RemoveContainer" containerID="7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7"
Oct 29 23:04:48.490408 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:04:48.490383    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7\": container with ID starting with 7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7 not found: ID does not exist" containerID="7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7"
Oct 29 23:04:48.490536 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:48.490411    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7"} err="failed to get container status \"7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7\": rpc error: code = NotFound desc = could not find container \"7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7\": container with ID starting with 7f6a2d3e8a14c414d82770efe96a193ad41c65ff7ae157f14fa8a76e398887f7 not found: ID does not exist"
Oct 29 23:04:48.543473 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:48.543449    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"]
Oct 29 23:04:48.556708 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:48.556687    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d68c2c-664fb4b5c-v7nq5"]
Oct 29 23:04:48.575480 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:48.575457    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:49.796853 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:49.796824    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:50.112228 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:50.112154    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2604c6c-436e-4aa2-9ec6-fe306749909d" path="/var/lib/kubelet/pods/b2604c6c-436e-4aa2-9ec6-fe306749909d/volumes"
Oct 29 23:04:51.632123 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:51.632088    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:52.896261 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:52.896228    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:53.021829 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:53.021798    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:53.146727 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:53.146627    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:54.367247 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:54.367208    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:54.491969 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:54.491932    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:54.651422 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:54.651307    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:54.654449 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:54.654423    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"]
Oct 29 23:04:54.654647 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:54.654621    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7" podUID="1ee7a61c-5560-4470-8f87-04361f3c4804" containerName="istio-proxy" containerID="cri-o://8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de" gracePeriod=30
Oct 29 23:04:54.668961 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:54.668935    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"]
Oct 29 23:04:54.669182 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:54.669160    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c" podUID="c0d6d3a9-38ae-481f-827b-80d1f3524ab6" containerName="istio-proxy" containerID="cri-o://30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62" gracePeriod=30
Oct 29 23:04:54.784010 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:54.783968    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:55.172138 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:55.172102    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:56.460051 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:56.460014    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:57.944981 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:57.944948    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:59.559677 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:59.559644    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:59.926365 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:59.926340    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 23:04:59.947510 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:04:59.947486    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 23:05:00.049208 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049172    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-podinfo\") pod \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") "
Oct 29 23:05:00.049424 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049224    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-data\") pod \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") "
Oct 29 23:05:00.049424 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049275    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-credential-socket\") pod \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") "
Oct 29 23:05:00.049424 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049309    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-data\") pod \"1ee7a61c-5560-4470-8f87-04361f3c4804\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") "
Oct 29 23:05:00.049424 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049382    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-token\" (UniqueName: \"kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-token\") pod \"1ee7a61c-5560-4470-8f87-04361f3c4804\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") "
Oct 29 23:05:00.049424 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049420    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istiod-ca-cert\") pod \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") "
Oct 29 23:05:00.049815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049474    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-socket\") pod \"1ee7a61c-5560-4470-8f87-04361f3c4804\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") "
Oct 29 23:05:00.049815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049503    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-socket\") pod \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") "
Oct 29 23:05:00.049815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049534    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-envoy\") pod \"1ee7a61c-5560-4470-8f87-04361f3c4804\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") "
Oct 29 23:05:00.049815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049574    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-kube-api-access-jvgsg\" (UniqueName: \"kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-kube-api-access-jvgsg\") pod \"1ee7a61c-5560-4470-8f87-04361f3c4804\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") "
Oct 29 23:05:00.049815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049616    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-data" pod "c0d6d3a9-38ae-481f-827b-80d1f3524ab6" (UID: "c0d6d3a9-38ae-481f-827b-80d1f3524ab6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.049815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049667    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-certs\") pod \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") "
Oct 29 23:05:00.049815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049698    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-certs\") pod \"1ee7a61c-5560-4470-8f87-04361f3c4804\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") "
Oct 29 23:05:00.049815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049726    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-token\") pod \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") "
Oct 29 23:05:00.049815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049764    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee7a61c-5560-4470-8f87-04361f3c4804-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee7a61c-5560-4470-8f87-04361f3c4804-istiod-ca-cert\") pod \"1ee7a61c-5560-4470-8f87-04361f3c4804\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") "
Oct 29 23:05:00.049815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049808    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-socket" pod "c0d6d3a9-38ae-481f-827b-80d1f3524ab6" (UID: "c0d6d3a9-38ae-481f-827b-80d1f3524ab6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.050416 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049833    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-podinfo\") pod \"1ee7a61c-5560-4470-8f87-04361f3c4804\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") "
Oct 29 23:05:00.050416 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049868    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-envoy\") pod \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") "
Oct 29 23:05:00.050416 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049899    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-kube-api-access-ss4hv\" (UniqueName: \"kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-kube-api-access-ss4hv\") pod \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\" (UID: \"c0d6d3a9-38ae-481f-827b-80d1f3524ab6\") "
Oct 29 23:05:00.050416 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049943    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-credential-socket\") pod \"1ee7a61c-5560-4470-8f87-04361f3c4804\" (UID: \"1ee7a61c-5560-4470-8f87-04361f3c4804\") "
Oct 29 23:05:00.050416 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.049991    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-socket" pod "1ee7a61c-5560-4470-8f87-04361f3c4804" (UID: "1ee7a61c-5560-4470-8f87-04361f3c4804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.050416 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.050366    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.050416 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.050385    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.050801 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.050423    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee7a61c-5560-4470-8f87-04361f3c4804-istiod-ca-cert" pod "1ee7a61c-5560-4470-8f87-04361f3c4804" (UID: "1ee7a61c-5560-4470-8f87-04361f3c4804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:00.050801 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.050504    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istiod-ca-cert" pod "c0d6d3a9-38ae-481f-827b-80d1f3524ab6" (UID: "c0d6d3a9-38ae-481f-827b-80d1f3524ab6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:00.050801 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.050649    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-credential-socket" pod "1ee7a61c-5560-4470-8f87-04361f3c4804" (UID: "1ee7a61c-5560-4470-8f87-04361f3c4804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.050990 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.050842    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-data" pod "1ee7a61c-5560-4470-8f87-04361f3c4804" (UID: "1ee7a61c-5560-4470-8f87-04361f3c4804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.051472 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.051444    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-certs" pod "1ee7a61c-5560-4470-8f87-04361f3c4804" (UID: "1ee7a61c-5560-4470-8f87-04361f3c4804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.051659 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.051636    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-certs" pod "c0d6d3a9-38ae-481f-827b-80d1f3524ab6" (UID: "c0d6d3a9-38ae-481f-827b-80d1f3524ab6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.052158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.052130    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-credential-socket" pod "c0d6d3a9-38ae-481f-827b-80d1f3524ab6" (UID: "c0d6d3a9-38ae-481f-827b-80d1f3524ab6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.052264 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.052198    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-podinfo" pod "c0d6d3a9-38ae-481f-827b-80d1f3524ab6" (UID: "c0d6d3a9-38ae-481f-827b-80d1f3524ab6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:05:00.053188 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.053158    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-podinfo" pod "1ee7a61c-5560-4470-8f87-04361f3c4804" (UID: "1ee7a61c-5560-4470-8f87-04361f3c4804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:05:00.054505 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.054479    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-envoy" pod "c0d6d3a9-38ae-481f-827b-80d1f3524ab6" (UID: "c0d6d3a9-38ae-481f-827b-80d1f3524ab6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.054612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.054582    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-kube-api-access-jvgsg" pod "1ee7a61c-5560-4470-8f87-04361f3c4804" (UID: "1ee7a61c-5560-4470-8f87-04361f3c4804"). InnerVolumeSpecName "kube-api-access-jvgsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:00.054726 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.054688    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-token" pod "1ee7a61c-5560-4470-8f87-04361f3c4804" (UID: "1ee7a61c-5560-4470-8f87-04361f3c4804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:00.055007 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.054980    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-envoy" pod "1ee7a61c-5560-4470-8f87-04361f3c4804" (UID: "1ee7a61c-5560-4470-8f87-04361f3c4804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:00.055651 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.055630    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-token" pod "c0d6d3a9-38ae-481f-827b-80d1f3524ab6" (UID: "c0d6d3a9-38ae-481f-827b-80d1f3524ab6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:00.056022 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.056004    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-kube-api-access-ss4hv" pod "c0d6d3a9-38ae-481f-827b-80d1f3524ab6" (UID: "c0d6d3a9-38ae-481f-827b-80d1f3524ab6"). InnerVolumeSpecName "kube-api-access-ss4hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:00.111749 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.111687    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"]
Oct 29 23:05:00.119771 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.119743    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"]
Oct 29 23:05:00.129070 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.129042    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"]
Oct 29 23:05:00.136500 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.136475    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"]
Oct 29 23:05:00.151820 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151796    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151820 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151821    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151830    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151839    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151847    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151858    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvgsg\" (UniqueName: \"kubernetes.io/projected/1ee7a61c-5560-4470-8f87-04361f3c4804-kube-api-access-jvgsg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151867    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151876    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151884    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151893    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee7a61c-5560-4470-8f87-04361f3c4804-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151900    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee7a61c-5560-4470-8f87-04361f3c4804-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151907    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151915    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss4hv\" (UniqueName: \"kubernetes.io/projected/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-kube-api-access-ss4hv\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151922    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee7a61c-5560-4470-8f87-04361f3c4804-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151930    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.151971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.151938    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d6d3a9-38ae-481f-827b-80d1f3524ab6-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:00.245131 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.245087    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:00.556528 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.556498    4860 generic.go:356] "Generic (PLEG): container finished" podID="1ee7a61c-5560-4470-8f87-04361f3c4804" containerID="8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de" exitCode=0
Oct 29 23:05:00.556723 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.556546    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"
Oct 29 23:05:00.556723 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.556574    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7" event={"ID":"1ee7a61c-5560-4470-8f87-04361f3c4804","Type":"ContainerDied","Data":"8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de"}
Oct 29 23:05:00.556723 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.556627    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7" event={"ID":"1ee7a61c-5560-4470-8f87-04361f3c4804","Type":"ContainerDied","Data":"7cbea702ff6bc3d0eb64180f15e3516d123a7c0cd25f37c1b8b84e4744e2a14b"}
Oct 29 23:05:00.556723 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.556652    4860 scope.go:122] "RemoveContainer" containerID="8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de"
Oct 29 23:05:00.558259 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.558237    4860 generic.go:356] "Generic (PLEG): container finished" podID="c0d6d3a9-38ae-481f-827b-80d1f3524ab6" containerID="30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62" exitCode=0
Oct 29 23:05:00.558425 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.558281    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"
Oct 29 23:05:00.558425 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.558357    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c" event={"ID":"c0d6d3a9-38ae-481f-827b-80d1f3524ab6","Type":"ContainerDied","Data":"30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62"}
Oct 29 23:05:00.558425 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.558405    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c" event={"ID":"c0d6d3a9-38ae-481f-827b-80d1f3524ab6","Type":"ContainerDied","Data":"5f9ed791b36de07a9ec40b911c09d76999db9738ff9bc02bc15fd98029afdfa5"}
Oct 29 23:05:00.567277 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.567254    4860 scope.go:122] "RemoveContainer" containerID="8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de"
Oct 29 23:05:00.567602 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:05:00.567553    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de\": container with ID starting with 8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de not found: ID does not exist" containerID="8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de"
Oct 29 23:05:00.567602 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.567579    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de"} err="failed to get container status \"8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de\": rpc error: code = NotFound desc = could not find container \"8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de\": container with ID starting with 8d60a747b7b6e000b7ef5c8609499d38b2fa195384e22adeea40fb62accc06de not found: ID does not exist"
Oct 29 23:05:00.567602 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.567595    4860 scope.go:122] "RemoveContainer" containerID="30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62"
Oct 29 23:05:00.576480 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.576460    4860 scope.go:122] "RemoveContainer" containerID="30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62"
Oct 29 23:05:00.576733 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:05:00.576714    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62\": container with ID starting with 30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62 not found: ID does not exist" containerID="30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62"
Oct 29 23:05:00.576797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.576740    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62"} err="failed to get container status \"30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62\": rpc error: code = NotFound desc = could not find container \"30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62\": container with ID starting with 30991475164c29ffa54b5e3817aeac1e3536136dfe9dd811cf464c94306f9c62 not found: ID does not exist"
Oct 29 23:05:00.614105 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.614084    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"]
Oct 29 23:05:00.622251 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.622233    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-xpz6c"]
Oct 29 23:05:00.656380 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.656358    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"]
Oct 29 23:05:00.662157 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:00.662139    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d68c2c-openshift-default-67bfdff69f-j4gn7"]
Oct 29 23:05:01.161438 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:01.161402    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:02.112136 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:02.112102    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee7a61c-5560-4470-8f87-04361f3c4804" path="/var/lib/kubelet/pods/1ee7a61c-5560-4470-8f87-04361f3c4804/volumes"
Oct 29 23:05:02.112649 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:02.112631    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d6d3a9-38ae-481f-827b-80d1f3524ab6" path="/var/lib/kubelet/pods/c0d6d3a9-38ae-481f-827b-80d1f3524ab6/volumes"
Oct 29 23:05:02.618835 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:02.618806    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:04.155820 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:04.155789    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:04.366959 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:04.366924    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"]
Oct 29 23:05:04.367199 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:04.367168    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy" containerID="cri-o://6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea" gracePeriod=30
Oct 29 23:05:05.655731 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:05.655694    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:05.788425 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:05.788390    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"]
Oct 29 23:05:05.914141 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:05.914073    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:06.035651 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:06.035620    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:07.229418 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:07.229384    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:08.500951 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.500915    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"]
Oct 29 23:05:08.501371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501319    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy"
Oct 29 23:05:08.501371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501344    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy"
Oct 29 23:05:08.501371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501353    4860 cpu_manager.go:414] "containerMap: removing container" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy"
Oct 29 23:05:08.501371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501358    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy"
Oct 29 23:05:08.501371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501370    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d13f5a1b-9637-4492-81c3-874e6c91294d" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501375    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13f5a1b-9637-4492-81c3-874e6c91294d" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501384    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ecb268da-f8b2-4be8-b379-f96e4eb818fc" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501389    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb268da-f8b2-4be8-b379-f96e4eb818fc" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501397    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1ee7a61c-5560-4470-8f87-04361f3c4804" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501402    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee7a61c-5560-4470-8f87-04361f3c4804" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501410    4860 cpu_manager.go:414] "containerMap: removing container" podUID="b2604c6c-436e-4aa2-9ec6-fe306749909d" containerName="maas-api"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501415    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2604c6c-436e-4aa2-9ec6-fe306749909d" containerName="maas-api"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501422    4860 cpu_manager.go:414] "containerMap: removing container" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501427    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501438    4860 cpu_manager.go:414] "containerMap: removing container" podUID="23c21cd0-fd07-47f5-bd77-91cdb5155ac8" containerName="revoke-keys"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501443    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c21cd0-fd07-47f5-bd77-91cdb5155ac8" containerName="revoke-keys"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501456    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c0d6d3a9-38ae-481f-827b-80d1f3524ab6" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501461    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d6d3a9-38ae-481f-827b-80d1f3524ab6" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501472    4860 cpu_manager.go:414] "containerMap: removing container" podUID="50800f81-4679-4607-accf-3b3920cd73d5" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501477    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50800f81-4679-4607-accf-3b3920cd73d5" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501541    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501550    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ce07ff-10ea-4294-bd37-aa7a12d9f4d0" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501556    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13f5a1b-9637-4492-81c3-874e6c91294d" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501562    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb268da-f8b2-4be8-b379-f96e4eb818fc" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501569    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee7a61c-5560-4470-8f87-04361f3c4804" containerName="istio-proxy"
Oct 29 23:05:08.501565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501576    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2604c6c-436e-4aa2-9ec6-fe306749909d" containerName="maas-api"
Oct 29 23:05:08.502212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501582    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5837adb2-8517-4f79-97c0-81535c4a9f8a" containerName="main"
Oct 29 23:05:08.502212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501588    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c21cd0-fd07-47f5-bd77-91cdb5155ac8" containerName="revoke-keys"
Oct 29 23:05:08.502212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501596    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d6d3a9-38ae-481f-827b-80d1f3524ab6" containerName="istio-proxy"
Oct 29 23:05:08.502212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.501603    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="50800f81-4679-4607-accf-3b3920cd73d5" containerName="istio-proxy"
Oct 29 23:05:08.506761 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.506740    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:08.520305 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.520273    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7922x\" (UniqueName: \"kubernetes.io/projected/ee057ffa-942e-4c17-be28-ba1e52156461-kube-api-access-7922x\") pod \"maas-api-revoke-sub-cdbeb821-rbf8n\" (UID: \"ee057ffa-942e-4c17-be28-ba1e52156461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:08.520449 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.520369    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdbeb821-rbf8n\" (UID: \"ee057ffa-942e-4c17-be28-ba1e52156461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:08.521257 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.521232    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdbeb821-rbf8n\" (UID: \"ee057ffa-942e-4c17-be28-ba1e52156461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:08.522287 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.522264    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"]
Oct 29 23:05:08.605229 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.605193    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:05:08.605229 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.605205    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:05:08.605229 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.605213    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:05:08.605466 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.605270    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:05:08.622503 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.622482    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdbeb821-rbf8n\" (UID: \"ee057ffa-942e-4c17-be28-ba1e52156461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:08.622592 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.622547    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7922x\" (UniqueName: \"kubernetes.io/projected/ee057ffa-942e-4c17-be28-ba1e52156461-kube-api-access-7922x\") pod \"maas-api-revoke-sub-cdbeb821-rbf8n\" (UID: \"ee057ffa-942e-4c17-be28-ba1e52156461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:08.622592 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.622581    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdbeb821-rbf8n\" (UID: \"ee057ffa-942e-4c17-be28-ba1e52156461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:08.623203 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.623185    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdbeb821-rbf8n\" (UID: \"ee057ffa-942e-4c17-be28-ba1e52156461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:08.624904 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.624885    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdbeb821-rbf8n\" (UID: \"ee057ffa-942e-4c17-be28-ba1e52156461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:08.642114 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.642090    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7922x\" (UniqueName: \"kubernetes.io/projected/ee057ffa-942e-4c17-be28-ba1e52156461-kube-api-access-7922x\") pod \"maas-api-revoke-sub-cdbeb821-rbf8n\" (UID: \"ee057ffa-942e-4c17-be28-ba1e52156461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:08.821238 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:08.821184    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:09.119061 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:05:09.119031    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee057ffa_942e_4c17_be28_ba1e52156461.slice/crio-b4025c67d7d22a7b2ec2fb0adb424a7c64f84291bfc078510e5c04f8ba3addff WatchSource:0}: Error finding container b4025c67d7d22a7b2ec2fb0adb424a7c64f84291bfc078510e5c04f8ba3addff: Status 404 returned error can't find the container with id b4025c67d7d22a7b2ec2fb0adb424a7c64f84291bfc078510e5c04f8ba3addff
Oct 29 23:05:09.121050 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.121019    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"]
Oct 29 23:05:09.479402 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.479375    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 29 23:05:09.479540 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.479427    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 29 23:05:09.604313 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.604287    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 23:05:09.617087 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.617060    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n" event={"ID":"ee057ffa-942e-4c17-be28-ba1e52156461","Type":"ContainerStarted","Data":"67aeabac6abcaa676e90ff82bc59e9976ff734d124595c01b30942bbea24bce7"}
Oct 29 23:05:09.617207 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.617117    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n" event={"ID":"ee057ffa-942e-4c17-be28-ba1e52156461","Type":"ContainerStarted","Data":"b4025c67d7d22a7b2ec2fb0adb424a7c64f84291bfc078510e5c04f8ba3addff"}
Oct 29 23:05:09.618343 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.618308    4860 generic.go:356] "Generic (PLEG): container finished" podID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerID="6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea" exitCode=0
Oct 29 23:05:09.618440 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.618352    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"
Oct 29 23:05:09.618440 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.618351    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" event={"ID":"ac94c943-a3cf-4f8b-b653-eabafeff9c3d","Type":"ContainerDied","Data":"6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea"}
Oct 29 23:05:09.618440 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.618420    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq" event={"ID":"ac94c943-a3cf-4f8b-b653-eabafeff9c3d","Type":"ContainerDied","Data":"bddb144e6f8e40fd44374fa4994bb2307f5dc6685d360b5444389ebdaa37eeeb"}
Oct 29 23:05:09.618607 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.618451    4860 scope.go:122] "RemoveContainer" containerID="6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea"
Oct 29 23:05:09.628428 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.628412    4860 scope.go:122] "RemoveContainer" containerID="6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea"
Oct 29 23:05:09.628707 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:05:09.628687    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea\": container with ID starting with 6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea not found: ID does not exist" containerID="6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea"
Oct 29 23:05:09.628769 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.628715    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea"} err="failed to get container status \"6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea\": rpc error: code = NotFound desc = could not find container \"6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea\": container with ID starting with 6bfa04dc6a990cfcadbf7a64793a689d059a548fc8b67b7be5ac808c95f75dea not found: ID does not exist"
Oct 29 23:05:09.631985 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.631967    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-credential-socket\") pod \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") "
Oct 29 23:05:09.632060 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.632014    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-certs\") pod \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") "
Oct 29 23:05:09.632120 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.632079    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-socket\") pod \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") "
Oct 29 23:05:09.632120 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.632108    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-envoy\") pod \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") "
Oct 29 23:05:09.632243 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.632184    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-token\" (UniqueName: \"kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-token\") pod \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") "
Oct 29 23:05:09.632243 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.632235    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-kube-api-access-kwzrc\" (UniqueName: \"kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-kube-api-access-kwzrc\") pod \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") "
Oct 29 23:05:09.632370 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.632267    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-credential-socket" pod "ac94c943-a3cf-4f8b-b653-eabafeff9c3d" (UID: "ac94c943-a3cf-4f8b-b653-eabafeff9c3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:09.632427 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.632405    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-data\") pod \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") "
Oct 29 23:05:09.632427 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.632409    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-certs" pod "ac94c943-a3cf-4f8b-b653-eabafeff9c3d" (UID: "ac94c943-a3cf-4f8b-b653-eabafeff9c3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:09.632550 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.632424    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-socket" pod "ac94c943-a3cf-4f8b-b653-eabafeff9c3d" (UID: "ac94c943-a3cf-4f8b-b653-eabafeff9c3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:09.632721 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.632698    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-data" pod "ac94c943-a3cf-4f8b-b653-eabafeff9c3d" (UID: "ac94c943-a3cf-4f8b-b653-eabafeff9c3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:09.632812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.632738    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-podinfo\") pod \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") "
Oct 29 23:05:09.632812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.632782    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istiod-ca-cert\") pod \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\" (UID: \"ac94c943-a3cf-4f8b-b653-eabafeff9c3d\") "
Oct 29 23:05:09.633530 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.633191    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.633530 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.633218    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.633530 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.633235    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.633530 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.633250    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.633530 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.633362    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istiod-ca-cert" pod "ac94c943-a3cf-4f8b-b653-eabafeff9c3d" (UID: "ac94c943-a3cf-4f8b-b653-eabafeff9c3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:09.634723 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.634690    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-token" pod "ac94c943-a3cf-4f8b-b653-eabafeff9c3d" (UID: "ac94c943-a3cf-4f8b-b653-eabafeff9c3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:09.634840 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.634803    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-envoy" pod "ac94c943-a3cf-4f8b-b653-eabafeff9c3d" (UID: "ac94c943-a3cf-4f8b-b653-eabafeff9c3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:09.634840 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.634827    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-kube-api-access-kwzrc" pod "ac94c943-a3cf-4f8b-b653-eabafeff9c3d" (UID: "ac94c943-a3cf-4f8b-b653-eabafeff9c3d"). InnerVolumeSpecName "kube-api-access-kwzrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:09.635984 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.635965    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-podinfo" pod "ac94c943-a3cf-4f8b-b653-eabafeff9c3d" (UID: "ac94c943-a3cf-4f8b-b653-eabafeff9c3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:05:09.718922 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.718863    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n" podStartSLOduration=1.718850572 podStartE2EDuration="1.718850572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:09.718155179 +0000 UTC m=+3051.742931330" watchObservedRunningTime="2026-10-29 23:05:09.718850572 +0000 UTC m=+3051.743626703"
Oct 29 23:05:09.734253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.734223    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.734384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.734257    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.734384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.734272    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwzrc\" (UniqueName: \"kubernetes.io/projected/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-kube-api-access-kwzrc\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.734384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.734287    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.734384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.734301    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac94c943-a3cf-4f8b-b653-eabafeff9c3d-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.965501 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.965473    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"]
Oct 29 23:05:09.979075 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:09.979022    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cr7nq"]
Oct 29 23:05:10.111249 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:10.111218    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" path="/var/lib/kubelet/pods/ac94c943-a3cf-4f8b-b653-eabafeff9c3d/volumes"
Oct 29 23:05:10.626531 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:10.626496    4860 generic.go:356] "Generic (PLEG): container finished" podID="ee057ffa-942e-4c17-be28-ba1e52156461" containerID="67aeabac6abcaa676e90ff82bc59e9976ff734d124595c01b30942bbea24bce7" exitCode=0
Oct 29 23:05:10.626977 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:10.626583    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n" event={"ID":"ee057ffa-942e-4c17-be28-ba1e52156461","Type":"ContainerDied","Data":"67aeabac6abcaa676e90ff82bc59e9976ff734d124595c01b30942bbea24bce7"}
Oct 29 23:05:10.788914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:10.788878    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn" podUID="0c31608a-a11b-4927-ae4d-10c871e174e7" containerName="payload-pre-processing" containerID="cri-o://c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294" gracePeriod=25
Oct 29 23:05:11.769822 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:11.769795    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:11.855310 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:11.855284    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-openshift-ca\") pod \"ee057ffa-942e-4c17-be28-ba1e52156461\" (UID: \"ee057ffa-942e-4c17-be28-ba1e52156461\") "
Oct 29 23:05:11.855469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:11.855351    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-xks-ca\") pod \"ee057ffa-942e-4c17-be28-ba1e52156461\" (UID: \"ee057ffa-942e-4c17-be28-ba1e52156461\") "
Oct 29 23:05:11.855469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:11.855371    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee057ffa-942e-4c17-be28-ba1e52156461-kube-api-access-7922x\" (UniqueName: \"kubernetes.io/projected/ee057ffa-942e-4c17-be28-ba1e52156461-kube-api-access-7922x\") pod \"ee057ffa-942e-4c17-be28-ba1e52156461\" (UID: \"ee057ffa-942e-4c17-be28-ba1e52156461\") "
Oct 29 23:05:11.855922 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:11.855679    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-openshift-ca" pod "ee057ffa-942e-4c17-be28-ba1e52156461" (UID: "ee057ffa-942e-4c17-be28-ba1e52156461"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:11.857574 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:11.857550    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-xks-ca" pod "ee057ffa-942e-4c17-be28-ba1e52156461" (UID: "ee057ffa-942e-4c17-be28-ba1e52156461"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:11.857686 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:11.857605    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee057ffa-942e-4c17-be28-ba1e52156461-kube-api-access-7922x" pod "ee057ffa-942e-4c17-be28-ba1e52156461" (UID: "ee057ffa-942e-4c17-be28-ba1e52156461"). InnerVolumeSpecName "kube-api-access-7922x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:11.956151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:11.956124    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.956151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:11.956149    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee057ffa-942e-4c17-be28-ba1e52156461-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:11.956297 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:11.956158    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7922x\" (UniqueName: \"kubernetes.io/projected/ee057ffa-942e-4c17-be28-ba1e52156461-kube-api-access-7922x\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:12.640575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:12.640546    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"
Oct 29 23:05:12.640772 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:12.640580    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n" event={"ID":"ee057ffa-942e-4c17-be28-ba1e52156461","Type":"ContainerDied","Data":"b4025c67d7d22a7b2ec2fb0adb424a7c64f84291bfc078510e5c04f8ba3addff"}
Oct 29 23:05:12.640772 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:12.640624    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4025c67d7d22a7b2ec2fb0adb424a7c64f84291bfc078510e5c04f8ba3addff"
Oct 29 23:05:15.135431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.135392    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main" containerID="cri-o://a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd" gracePeriod=45
Oct 29 23:05:15.144748 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.144717    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main" containerID="cri-o://afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597" gracePeriod=45
Oct 29 23:05:15.149866 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.149834    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main" containerID="cri-o://66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb" gracePeriod=45
Oct 29 23:05:15.158698 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.158673    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main" containerID="cri-o://a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433" gracePeriod=45
Oct 29 23:05:15.445052 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.445021    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 23:05:15.464303 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.464276    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 23:05:15.468048 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.468027    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 23:05:15.471814 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.471789    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 23:05:15.487190 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.487168    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-model-cache\") pod \"332d879d-e9f8-4f25-b6ef-3de840b331fc\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") "
Oct 29 23:05:15.487284 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.487219    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-home\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-home\") pod \"332d879d-e9f8-4f25-b6ef-3de840b331fc\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") "
Oct 29 23:05:15.487284 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.487248    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/332d879d-e9f8-4f25-b6ef-3de840b331fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/332d879d-e9f8-4f25-b6ef-3de840b331fc-tls-certs\") pod \"332d879d-e9f8-4f25-b6ef-3de840b331fc\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") "
Oct 29 23:05:15.487405 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.487293    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-dshm\") pod \"332d879d-e9f8-4f25-b6ef-3de840b331fc\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") "
Oct 29 23:05:15.487405 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.487395    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-tmp-dir\") pod \"332d879d-e9f8-4f25-b6ef-3de840b331fc\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") "
Oct 29 23:05:15.487541 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.487516    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332d879d-e9f8-4f25-b6ef-3de840b331fc-kube-api-access-5m76m\" (UniqueName: \"kubernetes.io/projected/332d879d-e9f8-4f25-b6ef-3de840b331fc-kube-api-access-5m76m\") pod \"332d879d-e9f8-4f25-b6ef-3de840b331fc\" (UID: \"332d879d-e9f8-4f25-b6ef-3de840b331fc\") "
Oct 29 23:05:15.487814 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.487514    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-model-cache" pod "332d879d-e9f8-4f25-b6ef-3de840b331fc" (UID: "332d879d-e9f8-4f25-b6ef-3de840b331fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.487913 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.487525    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-home" pod "332d879d-e9f8-4f25-b6ef-3de840b331fc" (UID: "332d879d-e9f8-4f25-b6ef-3de840b331fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.487913 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.487722    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-tmp-dir" pod "332d879d-e9f8-4f25-b6ef-3de840b331fc" (UID: "332d879d-e9f8-4f25-b6ef-3de840b331fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.490059 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.490030    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/332d879d-e9f8-4f25-b6ef-3de840b331fc-tls-certs" pod "332d879d-e9f8-4f25-b6ef-3de840b331fc" (UID: "332d879d-e9f8-4f25-b6ef-3de840b331fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:15.490167 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.490061    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-dshm" pod "332d879d-e9f8-4f25-b6ef-3de840b331fc" (UID: "332d879d-e9f8-4f25-b6ef-3de840b331fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.490167 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.490070    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332d879d-e9f8-4f25-b6ef-3de840b331fc-kube-api-access-5m76m" pod "332d879d-e9f8-4f25-b6ef-3de840b331fc" (UID: "332d879d-e9f8-4f25-b6ef-3de840b331fc"). InnerVolumeSpecName "kube-api-access-5m76m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:15.581567 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.581532    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:15.588901 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.588878    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-model-cache\") pod \"25a9daa1-d068-4a2e-8582-323349242bc5\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") "
Oct 29 23:05:15.589006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.588910    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-home\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-home\") pod \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") "
Oct 29 23:05:15.589006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.588941    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-kube-api-access-7bhjp\" (UniqueName: \"kubernetes.io/projected/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-kube-api-access-7bhjp\") pod \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") "
Oct 29 23:05:15.589006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.588991    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a78176-4066-475f-bb02-5f2ea982ebae-kube-api-access-jkzw4\" (UniqueName: \"kubernetes.io/projected/d2a78176-4066-475f-bb02-5f2ea982ebae-kube-api-access-jkzw4\") pod \"d2a78176-4066-475f-bb02-5f2ea982ebae\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") "
Oct 29 23:05:15.589175 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589019    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a78176-4066-475f-bb02-5f2ea982ebae-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a78176-4066-475f-bb02-5f2ea982ebae-tls-certs\") pod \"d2a78176-4066-475f-bb02-5f2ea982ebae\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") "
Oct 29 23:05:15.589175 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589048    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-model-cache\") pod \"d2a78176-4066-475f-bb02-5f2ea982ebae\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") "
Oct 29 23:05:15.589291 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589175    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-model-cache" pod "25a9daa1-d068-4a2e-8582-323349242bc5" (UID: "25a9daa1-d068-4a2e-8582-323349242bc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.589291 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589207    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tls-certs\") pod \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") "
Oct 29 23:05:15.589291 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589256    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-home\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-home\") pod \"25a9daa1-d068-4a2e-8582-323349242bc5\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") "
Oct 29 23:05:15.589291 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589208    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-home" pod "5e8a4cf2-3f5f-4bff-960b-da8cfb664377" (UID: "5e8a4cf2-3f5f-4bff-960b-da8cfb664377"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.589533 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589297    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-home\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-home\") pod \"d2a78176-4066-475f-bb02-5f2ea982ebae\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") "
Oct 29 23:05:15.589533 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589298    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-model-cache" pod "d2a78176-4066-475f-bb02-5f2ea982ebae" (UID: "d2a78176-4066-475f-bb02-5f2ea982ebae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.589533 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589375    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a9daa1-d068-4a2e-8582-323349242bc5-kube-api-access-r6c7j\" (UniqueName: \"kubernetes.io/projected/25a9daa1-d068-4a2e-8582-323349242bc5-kube-api-access-r6c7j\") pod \"25a9daa1-d068-4a2e-8582-323349242bc5\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") "
Oct 29 23:05:15.589533 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589420    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-dshm\") pod \"25a9daa1-d068-4a2e-8582-323349242bc5\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") "
Oct 29 23:05:15.589533 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589473    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-dshm\") pod \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") "
Oct 29 23:05:15.589533 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589490    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-home" pod "25a9daa1-d068-4a2e-8582-323349242bc5" (UID: "25a9daa1-d068-4a2e-8582-323349242bc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.589533 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589512    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-tmp-dir\") pod \"25a9daa1-d068-4a2e-8582-323349242bc5\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") "
Oct 29 23:05:15.589910 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589549    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tmp-dir\") pod \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") "
Oct 29 23:05:15.589910 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589589    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-tmp-dir\") pod \"d2a78176-4066-475f-bb02-5f2ea982ebae\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") "
Oct 29 23:05:15.589910 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589689    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-dshm\") pod \"d2a78176-4066-475f-bb02-5f2ea982ebae\" (UID: \"d2a78176-4066-475f-bb02-5f2ea982ebae\") "
Oct 29 23:05:15.589910 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589727    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-model-cache\") pod \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\" (UID: \"5e8a4cf2-3f5f-4bff-960b-da8cfb664377\") "
Oct 29 23:05:15.589910 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589771    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25a9daa1-d068-4a2e-8582-323349242bc5-tls-certs\" (UniqueName: \"kubernetes.io/secret/25a9daa1-d068-4a2e-8582-323349242bc5-tls-certs\") pod \"25a9daa1-d068-4a2e-8582-323349242bc5\" (UID: \"25a9daa1-d068-4a2e-8582-323349242bc5\") "
Oct 29 23:05:15.589910 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589793    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-tmp-dir" pod "25a9daa1-d068-4a2e-8582-323349242bc5" (UID: "25a9daa1-d068-4a2e-8582-323349242bc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.590240 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.589988    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tmp-dir" pod "5e8a4cf2-3f5f-4bff-960b-da8cfb664377" (UID: "5e8a4cf2-3f5f-4bff-960b-da8cfb664377"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.590240 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590167    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-model-cache" pod "5e8a4cf2-3f5f-4bff-960b-da8cfb664377" (UID: "5e8a4cf2-3f5f-4bff-960b-da8cfb664377"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.590240 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590194    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.590240 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590220    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.590240 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590235    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.590545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590251    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.590545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590266    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.590545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590281    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.590545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590296    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m76m\" (UniqueName: \"kubernetes.io/projected/332d879d-e9f8-4f25-b6ef-3de840b331fc-kube-api-access-5m76m\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.590545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590312    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.590545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590343    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/332d879d-e9f8-4f25-b6ef-3de840b331fc-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.590545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590357    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.590545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590373    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.590545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.590387    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/332d879d-e9f8-4f25-b6ef-3de840b331fc-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.591254 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.591221    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-tmp-dir" pod "d2a78176-4066-475f-bb02-5f2ea982ebae" (UID: "d2a78176-4066-475f-bb02-5f2ea982ebae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.591379 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.591356    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-home" pod "d2a78176-4066-475f-bb02-5f2ea982ebae" (UID: "d2a78176-4066-475f-bb02-5f2ea982ebae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.592556 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.592520    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25a9daa1-d068-4a2e-8582-323349242bc5-tls-certs" pod "25a9daa1-d068-4a2e-8582-323349242bc5" (UID: "25a9daa1-d068-4a2e-8582-323349242bc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:15.592913 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.592844    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a9daa1-d068-4a2e-8582-323349242bc5-kube-api-access-r6c7j" pod "25a9daa1-d068-4a2e-8582-323349242bc5" (UID: "25a9daa1-d068-4a2e-8582-323349242bc5"). InnerVolumeSpecName "kube-api-access-r6c7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:15.592913 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.592880    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-kube-api-access-7bhjp" pod "5e8a4cf2-3f5f-4bff-960b-da8cfb664377" (UID: "5e8a4cf2-3f5f-4bff-960b-da8cfb664377"). InnerVolumeSpecName "kube-api-access-7bhjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:15.593123 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.593085    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-dshm" pod "25a9daa1-d068-4a2e-8582-323349242bc5" (UID: "25a9daa1-d068-4a2e-8582-323349242bc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.593275 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.593251    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tls-certs" pod "5e8a4cf2-3f5f-4bff-960b-da8cfb664377" (UID: "5e8a4cf2-3f5f-4bff-960b-da8cfb664377"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:15.593371 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.593316    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-dshm" pod "5e8a4cf2-3f5f-4bff-960b-da8cfb664377" (UID: "5e8a4cf2-3f5f-4bff-960b-da8cfb664377"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.593667 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.593633    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a78176-4066-475f-bb02-5f2ea982ebae-kube-api-access-jkzw4" pod "d2a78176-4066-475f-bb02-5f2ea982ebae" (UID: "d2a78176-4066-475f-bb02-5f2ea982ebae"). InnerVolumeSpecName "kube-api-access-jkzw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:15.594109 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.594087    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a78176-4066-475f-bb02-5f2ea982ebae-tls-certs" pod "d2a78176-4066-475f-bb02-5f2ea982ebae" (UID: "d2a78176-4066-475f-bb02-5f2ea982ebae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:15.594903 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.594880    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-dshm" pod "d2a78176-4066-475f-bb02-5f2ea982ebae" (UID: "d2a78176-4066-475f-bb02-5f2ea982ebae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:15.659922 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.659832    4860 generic.go:356] "Generic (PLEG): container finished" podID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerID="a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd" exitCode=0
Oct 29 23:05:15.659922 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.659873    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"
Oct 29 23:05:15.659922 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.659913    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" event={"ID":"332d879d-e9f8-4f25-b6ef-3de840b331fc","Type":"ContainerDied","Data":"a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd"}
Oct 29 23:05:15.660215 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.659960    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn" event={"ID":"332d879d-e9f8-4f25-b6ef-3de840b331fc","Type":"ContainerDied","Data":"8f7d67c73c06ff6d8c7080267a8700bf440860a43c3fd0d6f2eecb14c5a9543c"}
Oct 29 23:05:15.660215 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.659979    4860 scope.go:122] "RemoveContainer" containerID="a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd"
Oct 29 23:05:15.661489 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.661461    4860 generic.go:356] "Generic (PLEG): container finished" podID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerID="66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb" exitCode=0
Oct 29 23:05:15.661610 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.661505    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"
Oct 29 23:05:15.661610 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.661552    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" event={"ID":"d2a78176-4066-475f-bb02-5f2ea982ebae","Type":"ContainerDied","Data":"66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb"}
Oct 29 23:05:15.661610 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.661594    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz" event={"ID":"d2a78176-4066-475f-bb02-5f2ea982ebae","Type":"ContainerDied","Data":"9424e3196c9bb6d85cf4f73166cdb11ad9689a1baf0d94a3dccd537e98c0aea9"}
Oct 29 23:05:15.663203 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.663179    4860 generic.go:356] "Generic (PLEG): container finished" podID="25a9daa1-d068-4a2e-8582-323349242bc5" containerID="a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433" exitCode=0
Oct 29 23:05:15.663384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.663223    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"
Oct 29 23:05:15.663384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.663260    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" event={"ID":"25a9daa1-d068-4a2e-8582-323349242bc5","Type":"ContainerDied","Data":"a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433"}
Oct 29 23:05:15.663384 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.663299    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d" event={"ID":"25a9daa1-d068-4a2e-8582-323349242bc5","Type":"ContainerDied","Data":"435427e69ca600be5f924ddb5a9e06c1bb03d44aefbf397556cd5cc5b149af6c"}
Oct 29 23:05:15.665950 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.665930    4860 generic.go:356] "Generic (PLEG): container finished" podID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerID="afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597" exitCode=0
Oct 29 23:05:15.666054 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.665959    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" event={"ID":"5e8a4cf2-3f5f-4bff-960b-da8cfb664377","Type":"ContainerDied","Data":"afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597"}
Oct 29 23:05:15.666054 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.665986    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v" event={"ID":"5e8a4cf2-3f5f-4bff-960b-da8cfb664377","Type":"ContainerDied","Data":"0d1c0d50ea69d53fa8f370c6cf5a258105056f75406683e25a89b5c2433f6887"}
Oct 29 23:05:15.666181 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.666056    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"
Oct 29 23:05:15.675787 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.675756    4860 scope.go:122] "RemoveContainer" containerID="a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd"
Oct 29 23:05:15.676176 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:05:15.676073    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd\": container with ID starting with a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd not found: ID does not exist" containerID="a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd"
Oct 29 23:05:15.676176 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.676108    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd"} err="failed to get container status \"a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd\": rpc error: code = NotFound desc = could not find container \"a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd\": container with ID starting with a54d8dc815485acdf36c2871b7c713c133f2ec056c952e9ac8a9f0bce2a892fd not found: ID does not exist"
Oct 29 23:05:15.676176 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.676152    4860 scope.go:122] "RemoveContainer" containerID="66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb"
Oct 29 23:05:15.685728 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.685711    4860 scope.go:122] "RemoveContainer" containerID="66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb"
Oct 29 23:05:15.686041 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:05:15.686022    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb\": container with ID starting with 66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb not found: ID does not exist" containerID="66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb"
Oct 29 23:05:15.686104 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.686049    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb"} err="failed to get container status \"66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb\": rpc error: code = NotFound desc = could not find container \"66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb\": container with ID starting with 66c25d4693c135c24f96b4bcfe030760ea52b9be962d88a07ef662c3276be7cb not found: ID does not exist"
Oct 29 23:05:15.686104 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.686066    4860 scope.go:122] "RemoveContainer" containerID="a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433"
Oct 29 23:05:15.691223 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.691198    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.691223 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.691217    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6c7j\" (UniqueName: \"kubernetes.io/projected/25a9daa1-d068-4a2e-8582-323349242bc5-kube-api-access-r6c7j\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.691442 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.691227    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a9daa1-d068-4a2e-8582-323349242bc5-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.691442 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.691235    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.691442 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.691243    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.691442 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.691250    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2a78176-4066-475f-bb02-5f2ea982ebae-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.691442 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.691257    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.691442 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.691266    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a9daa1-d068-4a2e-8582-323349242bc5-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.691442 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.691274    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bhjp\" (UniqueName: \"kubernetes.io/projected/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-kube-api-access-7bhjp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.691442 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.691282    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkzw4\" (UniqueName: \"kubernetes.io/projected/d2a78176-4066-475f-bb02-5f2ea982ebae-kube-api-access-jkzw4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.691442 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.691290    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a78176-4066-475f-bb02-5f2ea982ebae-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.691442 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.691308    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e8a4cf2-3f5f-4bff-960b-da8cfb664377-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:15.694388 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.694372    4860 scope.go:122] "RemoveContainer" containerID="a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433"
Oct 29 23:05:15.694640 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:05:15.694622    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433\": container with ID starting with a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433 not found: ID does not exist" containerID="a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433"
Oct 29 23:05:15.694695 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.694646    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433"} err="failed to get container status \"a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433\": rpc error: code = NotFound desc = could not find container \"a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433\": container with ID starting with a648c4e7d3e830bafe7a5705011f19901462dc395d6c6750c19277373b259433 not found: ID does not exist"
Oct 29 23:05:15.694695 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.694662    4860 scope.go:122] "RemoveContainer" containerID="afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597"
Oct 29 23:05:15.702598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.702580    4860 scope.go:122] "RemoveContainer" containerID="afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597"
Oct 29 23:05:15.702842 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:05:15.702824    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597\": container with ID starting with afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597 not found: ID does not exist" containerID="afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597"
Oct 29 23:05:15.702898 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.702848    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597"} err="failed to get container status \"afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597\": rpc error: code = NotFound desc = could not find container \"afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597\": container with ID starting with afd82f245813d1cedfc54abdf2659aa07ee463520a81a5979a57ea1c5d623597 not found: ID does not exist"
Oct 29 23:05:15.743393 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.743370    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"]
Oct 29 23:05:15.746932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.746905    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/distinct-2-w1-fc684b-kserve-75c468ddd8-xkdqn"]
Oct 29 23:05:15.798451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.798424    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"]
Oct 29 23:05:15.801844 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.801819    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/simulator-w1-fc684b-kserve-78554888bf-8fc7d"]
Oct 29 23:05:15.836645 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.836615    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"]
Oct 29 23:05:15.839773 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.839753    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/distinct-w1-fc684b-kserve-599d8c66bc-htj6v"]
Oct 29 23:05:15.874521 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.874491    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"]
Oct 29 23:05:15.878567 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:15.878546    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fc684b/premium-w1-fc684b-kserve-5c4df557b-6bwkz"]
Oct 29 23:05:16.110567 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:16.110535    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" path="/var/lib/kubelet/pods/25a9daa1-d068-4a2e-8582-323349242bc5/volumes"
Oct 29 23:05:16.111032 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:16.111010    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" path="/var/lib/kubelet/pods/332d879d-e9f8-4f25-b6ef-3de840b331fc/volumes"
Oct 29 23:05:16.111473 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:16.111454    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" path="/var/lib/kubelet/pods/5e8a4cf2-3f5f-4bff-960b-da8cfb664377/volumes"
Oct 29 23:05:16.112176 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:16.112155    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" path="/var/lib/kubelet/pods/d2a78176-4066-475f-bb02-5f2ea982ebae/volumes"
Oct 29 23:05:16.242682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:16.242638    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:18.054426 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:05:18.054377    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:05:18.714120 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:18.714086    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn" podUID="0c31608a-a11b-4927-ae4d-10c871e174e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:05:23.749309 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:23.749274    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:24.091982 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:24.091903    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:24.216035 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:24.216006    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:24.682279 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:24.682246    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:25.181085 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:25.181057    4860 scope.go:122] "RemoveContainer" containerID="a6c2c9e7750b2061f47e413ae864063567ae4693fb1a8431cce01cef1666b8fe"
Oct 29 23:05:25.191965 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:25.191943    4860 scope.go:122] "RemoveContainer" containerID="24ba0c485f2ac97ffa6fbaba1015717a68b6d7812ea078e2c7065a7e6e8cb650"
Oct 29 23:05:25.200187 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:25.200168    4860 scope.go:122] "RemoveContainer" containerID="e8eefd8ecc9da1186c30762eeb3645a3a6ae2b4f19557d5467efec8478142c8d"
Oct 29 23:05:25.208283 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:25.208266    4860 scope.go:122] "RemoveContainer" containerID="7b1e564ecdd5cde0cecf13c65ac17ce545abee7a3d31f48d6585054419b946db"
Oct 29 23:05:28.714387 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:28.714349    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn" podUID="0c31608a-a11b-4927-ae4d-10c871e174e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:05:30.543707 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:30.543667    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:30.667161 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:30.667131    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:30.770566 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:30.770533    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"]
Oct 29 23:05:30.770778 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:30.770752    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy" containerID="cri-o://7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece" gracePeriod=30
Oct 29 23:05:30.836684 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:30.836658    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:31.046571 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.046543    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"
Oct 29 23:05:31.129660 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.129592    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c31608a-a11b-4927-ae4d-10c871e174e7-kube-api-access-kj5vz\" (UniqueName: \"kubernetes.io/projected/0c31608a-a11b-4927-ae4d-10c871e174e7-kube-api-access-kj5vz\") pod \"0c31608a-a11b-4927-ae4d-10c871e174e7\" (UID: \"0c31608a-a11b-4927-ae4d-10c871e174e7\") "
Oct 29 23:05:31.129811 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.129675    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c31608a-a11b-4927-ae4d-10c871e174e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c31608a-a11b-4927-ae4d-10c871e174e7-plugins-config-volume\") pod \"0c31608a-a11b-4927-ae4d-10c871e174e7\" (UID: \"0c31608a-a11b-4927-ae4d-10c871e174e7\") "
Oct 29 23:05:31.130020 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.129999    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c31608a-a11b-4927-ae4d-10c871e174e7-plugins-config-volume" pod "0c31608a-a11b-4927-ae4d-10c871e174e7" (UID: "0c31608a-a11b-4927-ae4d-10c871e174e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:31.131714 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.131694    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c31608a-a11b-4927-ae4d-10c871e174e7-kube-api-access-kj5vz" pod "0c31608a-a11b-4927-ae4d-10c871e174e7" (UID: "0c31608a-a11b-4927-ae4d-10c871e174e7"). InnerVolumeSpecName "kube-api-access-kj5vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:31.231282 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.231256    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c31608a-a11b-4927-ae4d-10c871e174e7-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:31.231282 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.231278    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj5vz\" (UniqueName: \"kubernetes.io/projected/0c31608a-a11b-4927-ae4d-10c871e174e7-kube-api-access-kj5vz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:31.776509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.776476    4860 generic.go:356] "Generic (PLEG): container finished" podID="0c31608a-a11b-4927-ae4d-10c871e174e7" containerID="c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294" exitCode=0
Oct 29 23:05:31.777006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.776532    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"
Oct 29 23:05:31.777006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.776562    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn" event={"ID":"0c31608a-a11b-4927-ae4d-10c871e174e7","Type":"ContainerDied","Data":"c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294"}
Oct 29 23:05:31.777006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.776658    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn" event={"ID":"0c31608a-a11b-4927-ae4d-10c871e174e7","Type":"ContainerDied","Data":"cca9c7c53ab4e0c4ad8aa8ac0dc1a3407965f1250098fb998ac71f5e87baa10e"}
Oct 29 23:05:31.777006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.776694    4860 scope.go:122] "RemoveContainer" containerID="c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294"
Oct 29 23:05:31.787288 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.787272    4860 scope.go:122] "RemoveContainer" containerID="c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294"
Oct 29 23:05:31.787548 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:05:31.787532    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294\": container with ID starting with c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294 not found: ID does not exist" containerID="c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294"
Oct 29 23:05:31.787594 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.787555    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294"} err="failed to get container status \"c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294\": rpc error: code = NotFound desc = could not find container \"c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294\": container with ID starting with c895409382c3592e972dc2b759d1e459d00f5efec6b078398fcb5dde45180294 not found: ID does not exist"
Oct 29 23:05:31.830511 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.830488    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"]
Oct 29 23:05:31.843663 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:31.843634    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fc684b-7fb5fd79c5-hb2rn"]
Oct 29 23:05:32.110737 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:32.110666    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c31608a-a11b-4927-ae4d-10c871e174e7" path="/var/lib/kubelet/pods/0c31608a-a11b-4927-ae4d-10c871e174e7/volumes"
Oct 29 23:05:36.011939 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.011915    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 23:05:36.175073 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.174999    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-istio-token\" (UniqueName: \"kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-istio-token\") pod \"da28397d-fefc-4a03-8dc8-3fad87558566\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") "
Oct 29 23:05:36.175073 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175056    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-credential-socket\") pod \"da28397d-fefc-4a03-8dc8-3fad87558566\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") "
Oct 29 23:05:36.175300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175084    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-data\") pod \"da28397d-fefc-4a03-8dc8-3fad87558566\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") "
Oct 29 23:05:36.175300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175118    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-envoy\") pod \"da28397d-fefc-4a03-8dc8-3fad87558566\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") "
Oct 29 23:05:36.175300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175142    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-kube-api-access-b4h6t\" (UniqueName: \"kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-kube-api-access-b4h6t\") pod \"da28397d-fefc-4a03-8dc8-3fad87558566\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") "
Oct 29 23:05:36.175300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175193    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-socket\") pod \"da28397d-fefc-4a03-8dc8-3fad87558566\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") "
Oct 29 23:05:36.175300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175233    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da28397d-fefc-4a03-8dc8-3fad87558566-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da28397d-fefc-4a03-8dc8-3fad87558566-istiod-ca-cert\") pod \"da28397d-fefc-4a03-8dc8-3fad87558566\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") "
Oct 29 23:05:36.175300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175274    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da28397d-fefc-4a03-8dc8-3fad87558566-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da28397d-fefc-4a03-8dc8-3fad87558566-istio-podinfo\") pod \"da28397d-fefc-4a03-8dc8-3fad87558566\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") "
Oct 29 23:05:36.175681 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175358    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-certs\") pod \"da28397d-fefc-4a03-8dc8-3fad87558566\" (UID: \"da28397d-fefc-4a03-8dc8-3fad87558566\") "
Oct 29 23:05:36.175681 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175467    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-data" pod "da28397d-fefc-4a03-8dc8-3fad87558566" (UID: "da28397d-fefc-4a03-8dc8-3fad87558566"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:36.175681 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175485    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-credential-socket" pod "da28397d-fefc-4a03-8dc8-3fad87558566" (UID: "da28397d-fefc-4a03-8dc8-3fad87558566"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:36.175681 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175656    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-certs" pod "da28397d-fefc-4a03-8dc8-3fad87558566" (UID: "da28397d-fefc-4a03-8dc8-3fad87558566"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:36.175919 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175698    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da28397d-fefc-4a03-8dc8-3fad87558566-istiod-ca-cert" pod "da28397d-fefc-4a03-8dc8-3fad87558566" (UID: "da28397d-fefc-4a03-8dc8-3fad87558566"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:36.175919 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175751    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.175919 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175770    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.175919 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175784    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.175919 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.175873    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-socket" pod "da28397d-fefc-4a03-8dc8-3fad87558566" (UID: "da28397d-fefc-4a03-8dc8-3fad87558566"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:36.177626 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.177603    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da28397d-fefc-4a03-8dc8-3fad87558566-istio-podinfo" pod "da28397d-fefc-4a03-8dc8-3fad87558566" (UID: "da28397d-fefc-4a03-8dc8-3fad87558566"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:05:36.177626 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.177615    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-kube-api-access-b4h6t" pod "da28397d-fefc-4a03-8dc8-3fad87558566" (UID: "da28397d-fefc-4a03-8dc8-3fad87558566"). InnerVolumeSpecName "kube-api-access-b4h6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:36.177826 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.177778    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-istio-token" pod "da28397d-fefc-4a03-8dc8-3fad87558566" (UID: "da28397d-fefc-4a03-8dc8-3fad87558566"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:36.177826 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.177779    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-envoy" pod "da28397d-fefc-4a03-8dc8-3fad87558566" (UID: "da28397d-fefc-4a03-8dc8-3fad87558566"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:05:36.277278 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.277193    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.277278 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.277226    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da28397d-fefc-4a03-8dc8-3fad87558566-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.277278 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.277241    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da28397d-fefc-4a03-8dc8-3fad87558566-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.277278 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.277256    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.277278 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.277268    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da28397d-fefc-4a03-8dc8-3fad87558566-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.277278 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.277280    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4h6t\" (UniqueName: \"kubernetes.io/projected/da28397d-fefc-4a03-8dc8-3fad87558566-kube-api-access-b4h6t\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:05:36.602785 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.602742    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:36.719071 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.719018    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:36.807526 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.807498    4860 generic.go:356] "Generic (PLEG): container finished" podID="da28397d-fefc-4a03-8dc8-3fad87558566" containerID="7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece" exitCode=0
Oct 29 23:05:36.807686 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.807549    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"
Oct 29 23:05:36.807686 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.807581    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" event={"ID":"da28397d-fefc-4a03-8dc8-3fad87558566","Type":"ContainerDied","Data":"7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece"}
Oct 29 23:05:36.807686 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.807634    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz" event={"ID":"da28397d-fefc-4a03-8dc8-3fad87558566","Type":"ContainerDied","Data":"d6c5c65696e047cc92ee428c15a77b25e1650f9166c35b9a61c3574853c42a6b"}
Oct 29 23:05:36.807686 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.807656    4860 scope.go:122] "RemoveContainer" containerID="7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece"
Oct 29 23:05:36.817556 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.817541    4860 scope.go:122] "RemoveContainer" containerID="ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61"
Oct 29 23:05:36.829374 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.829349    4860 scope.go:122] "RemoveContainer" containerID="7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece"
Oct 29 23:05:36.829673 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:05:36.829655    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece\": container with ID starting with 7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece not found: ID does not exist" containerID="7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece"
Oct 29 23:05:36.829742 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.829684    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece"} err="failed to get container status \"7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece\": rpc error: code = NotFound desc = could not find container \"7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece\": container with ID starting with 7045894bb5b31c5356c94a0579118e84d8991892646e099a67e11737a92e7ece not found: ID does not exist"
Oct 29 23:05:36.829742 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.829700    4860 scope.go:122] "RemoveContainer" containerID="ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61"
Oct 29 23:05:36.829946 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:05:36.829933    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61\": container with ID starting with ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61 not found: ID does not exist" containerID="ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61"
Oct 29 23:05:36.829997 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.829950    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61"} err="failed to get container status \"ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61\": rpc error: code = NotFound desc = could not find container \"ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61\": container with ID starting with ee238dc6f4970723a8badee5234af98fc5fd51190cef4ea5068badd2c1e08a61 not found: ID does not exist"
Oct 29 23:05:36.863044 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.862993    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"]
Oct 29 23:05:36.876333 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:36.876305    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fc684b-openshift-default-7fcf678997-988qz"]
Oct 29 23:05:37.627436 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:37.627398    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:38.110261 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:38.110226    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" path="/var/lib/kubelet/pods/da28397d-fefc-4a03-8dc8-3fad87558566/volumes"
Oct 29 23:05:43.208524 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:05:43.208482    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:48.055384 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:05:48.055344    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:06:04.540540 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540505    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"]
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540858    4860 cpu_manager.go:414] "containerMap: removing container" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540869    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540878    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540883    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540892    4860 cpu_manager.go:414] "containerMap: removing container" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540897    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540908    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540912    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540919    4860 cpu_manager.go:414] "containerMap: removing container" podUID="0c31608a-a11b-4927-ae4d-10c871e174e7" containerName="payload-pre-processing"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540925    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c31608a-a11b-4927-ae4d-10c871e174e7" containerName="payload-pre-processing"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540942    4860 cpu_manager.go:414] "containerMap: removing container" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540946    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540956    4860 cpu_manager.go:414] "containerMap: removing container" podUID="ee057ffa-942e-4c17-be28-ba1e52156461" containerName="revoke-keys"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540960    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee057ffa-942e-4c17-be28-ba1e52156461" containerName="revoke-keys"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540967    4860 cpu_manager.go:414] "containerMap: removing container" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540971    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540982    4860 cpu_manager.go:414] "containerMap: removing container" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.540986    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.541037    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="332d879d-e9f8-4f25-b6ef-3de840b331fc" containerName="main"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.541047    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a9daa1-d068-4a2e-8582-323349242bc5" containerName="main"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.541055    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.541061    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a78176-4066-475f-bb02-5f2ea982ebae" containerName="main"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.541067    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8a4cf2-3f5f-4bff-960b-da8cfb664377" containerName="main"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.541074    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac94c943-a3cf-4f8b-b653-eabafeff9c3d" containerName="istio-proxy"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.541079    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c31608a-a11b-4927-ae4d-10c871e174e7" containerName="payload-pre-processing"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.541087    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="da28397d-fefc-4a03-8dc8-3fad87558566" containerName="istio-proxy"
Oct 29 23:06:04.543058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.541093    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee057ffa-942e-4c17-be28-ba1e52156461" containerName="revoke-keys"
Oct 29 23:06:04.544194 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.544177    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.559532 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.559506    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-67ntj\""
Oct 29 23:06:04.575159 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.575135    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"]
Oct 29 23:06:04.605096 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.605071    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.605212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.605100    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.605212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.605124    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.605212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.605141    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.605212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.605199    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bbmx\" (UniqueName: \"kubernetes.io/projected/cc9cb970-f2d1-4e47-be8a-20e20982ff32-kube-api-access-7bbmx\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.605391 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.605276    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.605432 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.605389    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.605432 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.605414    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.605534 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.605520    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706011 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.705977    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706139 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706020    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706139 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706037    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706281 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706157    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706281 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706207    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706281 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706234    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bbmx\" (UniqueName: \"kubernetes.io/projected/cc9cb970-f2d1-4e47-be8a-20e20982ff32-kube-api-access-7bbmx\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706281 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706262    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706534 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706307    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706534 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706378    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706651 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706552    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706651 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706619    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706651 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706639    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706689    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.706812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.706769    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.708614 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.708593    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.708839 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.708820    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.726419 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.726392    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc9cb970-f2d1-4e47-be8a-20e20982ff32-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.726506 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.726444    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bbmx\" (UniqueName: \"kubernetes.io/projected/cc9cb970-f2d1-4e47-be8a-20e20982ff32-kube-api-access-7bbmx\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vkx5n\" (UID: \"cc9cb970-f2d1-4e47-be8a-20e20982ff32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:04.856610 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:04.856561    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:05.165424 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:06:05.165394    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9cb970_f2d1_4e47_be8a_20e20982ff32.slice/crio-03f19f62685074d69b3f2b3d1c48c491e8cabbed8de020f8d6956d5f265c082d WatchSource:0}: Error finding container 03f19f62685074d69b3f2b3d1c48c491e8cabbed8de020f8d6956d5f265c082d: Status 404 returned error can't find the container with id 03f19f62685074d69b3f2b3d1c48c491e8cabbed8de020f8d6956d5f265c082d
Oct 29 23:06:05.167354 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:05.167211    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"]
Oct 29 23:06:05.167903 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:05.167871    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:06:05.167977 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:05.167936    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:06:05.167977 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:05.167970    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:06:05.993071 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:05.993028    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n" event={"ID":"cc9cb970-f2d1-4e47-be8a-20e20982ff32","Type":"ContainerStarted","Data":"912d580a108a5f89374d4ce65554f2c32719cc51988af8588aff0a4549621493"}
Oct 29 23:06:05.993579 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:05.993111    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n" event={"ID":"cc9cb970-f2d1-4e47-be8a-20e20982ff32","Type":"ContainerStarted","Data":"03f19f62685074d69b3f2b3d1c48c491e8cabbed8de020f8d6956d5f265c082d"}
Oct 29 23:06:06.049848 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:06.049794    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n" podStartSLOduration=2.049776973 podStartE2EDuration="2.049776973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:06.04893872 +0000 UTC m=+3108.073714878" watchObservedRunningTime="2026-10-29 23:06:06.049776973 +0000 UTC m=+3108.074553105"
Oct 29 23:06:06.857065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:06.857031    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:06.858670 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:06.858643    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 29 23:06:06.858794 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:06.858689    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n" podUID="cc9cb970-f2d1-4e47-be8a-20e20982ff32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 29 23:06:07.857643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:07.857604    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 29 23:06:07.858036 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:07.857662    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n" podUID="cc9cb970-f2d1-4e47-be8a-20e20982ff32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 29 23:06:08.857023 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:08.856983    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 29 23:06:08.857205 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:08.857036    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n" podUID="cc9cb970-f2d1-4e47-be8a-20e20982ff32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 29 23:06:09.857003 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:09.856967    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 29 23:06:09.857459 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:09.857027    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n" podUID="cc9cb970-f2d1-4e47-be8a-20e20982ff32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 29 23:06:10.531569 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:10.531534    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:10.857879 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:10.857793    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 29 23:06:10.857879 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:10.857862    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n" podUID="cc9cb970-f2d1-4e47-be8a-20e20982ff32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 29 23:06:11.857029 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:11.856987    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused" start-of-body=
Oct 29 23:06:11.857211 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:11.857055    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n" podUID="cc9cb970-f2d1-4e47-be8a-20e20982ff32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": dial tcp 10.132.1.117:15021: connect: connection refused"
Oct 29 23:06:12.848889 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:12.848839    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:13.390856 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:13.390821    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:13.391277 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:13.391237    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:13.391966 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:13.391947    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n"
Oct 29 23:06:15.704725 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:15.704692    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:16.178150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:16.178078    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:18.054008 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:06:18.053970    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:06:19.582842 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.582810    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"]
Oct 29 23:06:19.586293 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.586268    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.617470 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.617446    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"]
Oct 29 23:06:19.633467 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.633442    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.633599 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.633485    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.633599 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.633505    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e280bf1d-cfc4-48ba-8531-720a62ebd898-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.633742 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.633632    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.633742 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.633701    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vcq\" (UniqueName: \"kubernetes.io/projected/e280bf1d-cfc4-48ba-8531-720a62ebd898-kube-api-access-n4vcq\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.633831 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.633755    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.633871 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.633822    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.633908 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.633881    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e280bf1d-cfc4-48ba-8531-720a62ebd898-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.633951 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.633917    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e280bf1d-cfc4-48ba-8531-720a62ebd898-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.734877 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.734843    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.734988 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.734895    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vcq\" (UniqueName: \"kubernetes.io/projected/e280bf1d-cfc4-48ba-8531-720a62ebd898-kube-api-access-n4vcq\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.734988 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.734926    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.734988 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.734944    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.734988 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.734971    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e280bf1d-cfc4-48ba-8531-720a62ebd898-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.734988 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.734989    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e280bf1d-cfc4-48ba-8531-720a62ebd898-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.735266 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.735060    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.735266 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.735128    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.735266 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.735166    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e280bf1d-cfc4-48ba-8531-720a62ebd898-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.735472 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.735311    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.735472 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.735366    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.735472 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.735457    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.735643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.735522    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.735708 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.735689    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e280bf1d-cfc4-48ba-8531-720a62ebd898-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.737595 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.737575    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e280bf1d-cfc4-48ba-8531-720a62ebd898-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.737689 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.737599    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e280bf1d-cfc4-48ba-8531-720a62ebd898-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.762134 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.762114    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e280bf1d-cfc4-48ba-8531-720a62ebd898-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.765309 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.765285    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vcq\" (UniqueName: \"kubernetes.io/projected/e280bf1d-cfc4-48ba-8531-720a62ebd898-kube-api-access-n4vcq\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzv89\" (UID: \"e280bf1d-cfc4-48ba-8531-720a62ebd898\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:19.899252 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:19.899202    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:20.207297 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:06:20.207272    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode280bf1d_cfc4_48ba_8531_720a62ebd898.slice/crio-9046cab0e8462e9ab68977ffc147bc81a281e8c06838e38f04fa30b5f809cb94 WatchSource:0}: Error finding container 9046cab0e8462e9ab68977ffc147bc81a281e8c06838e38f04fa30b5f809cb94: Status 404 returned error can't find the container with id 9046cab0e8462e9ab68977ffc147bc81a281e8c06838e38f04fa30b5f809cb94
Oct 29 23:06:20.208306 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:20.208284    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"]
Oct 29 23:06:20.209460 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:20.209426    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:06:20.209528 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:20.209491    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:06:20.209528 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:20.209520    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:06:21.097693 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:21.097651    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89" event={"ID":"e280bf1d-cfc4-48ba-8531-720a62ebd898","Type":"ContainerStarted","Data":"92422759afc400bdc91845526c46ec6a594e3628841c83c8985576233300e556"}
Oct 29 23:06:21.098235 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:21.097716    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89" event={"ID":"e280bf1d-cfc4-48ba-8531-720a62ebd898","Type":"ContainerStarted","Data":"9046cab0e8462e9ab68977ffc147bc81a281e8c06838e38f04fa30b5f809cb94"}
Oct 29 23:06:21.155837 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:21.155782    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89" podStartSLOduration=2.155768299 podStartE2EDuration="2.155768299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:21.155509877 +0000 UTC m=+3123.180286008" watchObservedRunningTime="2026-10-29 23:06:21.155768299 +0000 UTC m=+3123.180544514"
Oct 29 23:06:21.899481 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:21.899447    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:21.900602 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:21.900581    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 29 23:06:21.900664 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:21.900627    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89" podUID="e280bf1d-cfc4-48ba-8531-720a62ebd898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 29 23:06:22.899657 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:22.899620    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 29 23:06:22.900058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:22.899675    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89" podUID="e280bf1d-cfc4-48ba-8531-720a62ebd898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 29 23:06:23.899826 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:23.899783    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 29 23:06:23.900317 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:23.899847    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89" podUID="e280bf1d-cfc4-48ba-8531-720a62ebd898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 29 23:06:24.899695 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:24.899655    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 29 23:06:24.899895 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:24.899722    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89" podUID="e280bf1d-cfc4-48ba-8531-720a62ebd898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 29 23:06:25.292981 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:25.292948    4860 scope.go:122] "RemoveContainer" containerID="55076b5bdb1943b44c97f28727e06b247d4b05d896957c94d637493c36f0b69f"
Oct 29 23:06:25.899821 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:25.899786    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 29 23:06:25.900096 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:25.900072    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89" podUID="e280bf1d-cfc4-48ba-8531-720a62ebd898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 29 23:06:26.899497 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:26.899464    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 29 23:06:26.899679 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:26.899512    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89" podUID="e280bf1d-cfc4-48ba-8531-720a62ebd898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 29 23:06:28.172010 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:28.171972    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:28.285255 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:28.285219    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:28.421914 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:28.421878    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:28.422434 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:28.422345    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:28.423061 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:28.423041    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89"
Oct 29 23:06:35.028901 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:35.028870    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"]
Oct 29 23:06:35.041811 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:35.041783    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5478a7af-db96c"]
Oct 29 23:06:36.109410 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:06:36.109377    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa5e424-881b-483d-a583-5975412c2e06" path="/var/lib/kubelet/pods/4fa5e424-881b-483d-a583-5975412c2e06/volumes"
Oct 29 23:06:48.055655 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:06:48.055613    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:07:04.725316 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.725270    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"]
Oct 29 23:07:04.729954 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.729930    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.746758 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.746734    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"]
Oct 29 23:07:04.844126 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.844090    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.844304 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.844134    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.844304 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.844162    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j7mc\" (UniqueName: \"kubernetes.io/projected/58904265-85eb-43c4-9f23-dbaa1a682acc-kube-api-access-9j7mc\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.844304 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.844247    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58904265-85eb-43c4-9f23-dbaa1a682acc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.844521 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.844350    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.844521 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.844421    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58904265-85eb-43c4-9f23-dbaa1a682acc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.844521 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.844456    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.844521 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.844508    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58904265-85eb-43c4-9f23-dbaa1a682acc-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.844669 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.844538    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.944934 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.944902    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58904265-85eb-43c4-9f23-dbaa1a682acc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.945086 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.944940    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.945086 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.944970    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58904265-85eb-43c4-9f23-dbaa1a682acc-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.945086 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.945005    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.945265 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.945087    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.945265 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.945121    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.945265 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.945155    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j7mc\" (UniqueName: \"kubernetes.io/projected/58904265-85eb-43c4-9f23-dbaa1a682acc-kube-api-access-9j7mc\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.945488 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.945398    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.945552 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.945486    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.945552 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.945502    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58904265-85eb-43c4-9f23-dbaa1a682acc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.945673 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.945562    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.945673 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.945505    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.945897 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.945879    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.946001 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.945979    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58904265-85eb-43c4-9f23-dbaa1a682acc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.947492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.947468    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58904265-85eb-43c4-9f23-dbaa1a682acc-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.947693 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.947673    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58904265-85eb-43c4-9f23-dbaa1a682acc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.965584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.965557    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j7mc\" (UniqueName: \"kubernetes.io/projected/58904265-85eb-43c4-9f23-dbaa1a682acc-kube-api-access-9j7mc\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:04.966014 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:04.965985    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58904265-85eb-43c4-9f23-dbaa1a682acc-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-96ppp\" (UID: \"58904265-85eb-43c4-9f23-dbaa1a682acc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:05.046528 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:05.046463    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:05.387274 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:05.387233    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"]
Oct 29 23:07:05.390699 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:07:05.390664    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58904265_85eb_43c4_9f23_dbaa1a682acc.slice/crio-49d5b8833f06c91693d6906b60e2715466235c0539c1cde522b5a4cc0b74f6a3 WatchSource:0}: Error finding container 49d5b8833f06c91693d6906b60e2715466235c0539c1cde522b5a4cc0b74f6a3: Status 404 returned error can't find the container with id 49d5b8833f06c91693d6906b60e2715466235c0539c1cde522b5a4cc0b74f6a3
Oct 29 23:07:05.394119 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:05.394070    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:07:05.394233 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:05.394164    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:07:05.394233 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:05.394206    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:07:06.362340 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:06.362297    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp" event={"ID":"58904265-85eb-43c4-9f23-dbaa1a682acc","Type":"ContainerStarted","Data":"ea32388657991b3764babd364346b0303f1ba702457f7e5bf20c19f15a850d41"}
Oct 29 23:07:06.362856 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:06.362376    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp" event={"ID":"58904265-85eb-43c4-9f23-dbaa1a682acc","Type":"ContainerStarted","Data":"49d5b8833f06c91693d6906b60e2715466235c0539c1cde522b5a4cc0b74f6a3"}
Oct 29 23:07:06.421351 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:06.421278    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp" podStartSLOduration=2.421265991 podStartE2EDuration="2.421265991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:06.420764507 +0000 UTC m=+3168.445540653" watchObservedRunningTime="2026-10-29 23:07:06.421265991 +0000 UTC m=+3168.446042118"
Oct 29 23:07:07.046843 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:07.046813    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:07.048487 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:07.048463    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-96ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 29 23:07:07.048614 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:07.048510    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp" podUID="58904265-85eb-43c4-9f23-dbaa1a682acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 29 23:07:08.047650 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:08.047580    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-96ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 29 23:07:08.048169 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:08.047672    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp" podUID="58904265-85eb-43c4-9f23-dbaa1a682acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 29 23:07:09.047634 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:09.047597    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-96ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 29 23:07:09.047812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:09.047654    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp" podUID="58904265-85eb-43c4-9f23-dbaa1a682acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 29 23:07:10.046937 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:10.046899    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-96ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 29 23:07:10.047131 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:10.046952    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp" podUID="58904265-85eb-43c4-9f23-dbaa1a682acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 29 23:07:11.047270 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:11.047232    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-96ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 29 23:07:11.047706 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:11.047284    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp" podUID="58904265-85eb-43c4-9f23-dbaa1a682acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 29 23:07:12.046783 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:12.046748    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-96ppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 29 23:07:12.046975 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:12.046804    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp" podUID="58904265-85eb-43c4-9f23-dbaa1a682acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 29 23:07:13.678474 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:13.678439    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:13.678957 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:13.678891    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:13.679798 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:13.679778    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp"
Oct 29 23:07:15.820692 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:15.820656    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:16.277380 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:16.277346    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:18.054300 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:07:18.054267    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:07:48.056424 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:07:48.056312    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:07:50.232871 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:07:50.232840    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:13.509945 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.509912    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"]
Oct 29 23:08:13.510457 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.510086    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f" podUID="d5567035-eebe-4ab7-8875-876f89630445" containerName="discovery" containerID="cri-o://c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82" gracePeriod=30
Oct 29 23:08:13.786843 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.786779    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 23:08:13.852558 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.852520    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5567035-eebe-4ab7-8875-876f89630445-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5567035-eebe-4ab7-8875-876f89630445-local-certs\") pod \"d5567035-eebe-4ab7-8875-876f89630445\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") "
Oct 29 23:08:13.852767 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.852634    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-cacerts\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-cacerts\") pod \"d5567035-eebe-4ab7-8875-876f89630445\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") "
Oct 29 23:08:13.852767 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.852681    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-dns-cert\") pod \"d5567035-eebe-4ab7-8875-876f89630445\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") "
Oct 29 23:08:13.852767 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.852723    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-kubeconfig\") pod \"d5567035-eebe-4ab7-8875-876f89630445\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") "
Oct 29 23:08:13.852767 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.852762    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-ca-configmap\") pod \"d5567035-eebe-4ab7-8875-876f89630445\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") "
Oct 29 23:08:13.853046 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.852819    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-kube-api-access-kn8bs\" (UniqueName: \"kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-kube-api-access-kn8bs\") pod \"d5567035-eebe-4ab7-8875-876f89630445\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") "
Oct 29 23:08:13.853046 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.852856    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-istio-token\" (UniqueName: \"kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-istio-token\") pod \"d5567035-eebe-4ab7-8875-876f89630445\" (UID: \"d5567035-eebe-4ab7-8875-876f89630445\") "
Oct 29 23:08:13.853701 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.853657    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-ca-configmap" pod "d5567035-eebe-4ab7-8875-876f89630445" (UID: "d5567035-eebe-4ab7-8875-876f89630445"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:13.856002 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.855959    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-istio-token" pod "d5567035-eebe-4ab7-8875-876f89630445" (UID: "d5567035-eebe-4ab7-8875-876f89630445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:13.860562 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.860450    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-cacerts" pod "d5567035-eebe-4ab7-8875-876f89630445" (UID: "d5567035-eebe-4ab7-8875-876f89630445"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:13.860562 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.860480    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-kubeconfig" pod "d5567035-eebe-4ab7-8875-876f89630445" (UID: "d5567035-eebe-4ab7-8875-876f89630445"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:13.860562 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.860526    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-dns-cert" pod "d5567035-eebe-4ab7-8875-876f89630445" (UID: "d5567035-eebe-4ab7-8875-876f89630445"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:13.860809 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.860582    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-kube-api-access-kn8bs" pod "d5567035-eebe-4ab7-8875-876f89630445" (UID: "d5567035-eebe-4ab7-8875-876f89630445"). InnerVolumeSpecName "kube-api-access-kn8bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:13.860988 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.860952    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5567035-eebe-4ab7-8875-876f89630445-local-certs" pod "d5567035-eebe-4ab7-8875-876f89630445" (UID: "d5567035-eebe-4ab7-8875-876f89630445"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:08:13.871699 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.870629    4860 generic.go:356] "Generic (PLEG): container finished" podID="d5567035-eebe-4ab7-8875-876f89630445" containerID="c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82" exitCode=0
Oct 29 23:08:13.871699 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.870713    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f" event={"ID":"d5567035-eebe-4ab7-8875-876f89630445","Type":"ContainerDied","Data":"c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82"}
Oct 29 23:08:13.871699 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.870764    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f" event={"ID":"d5567035-eebe-4ab7-8875-876f89630445","Type":"ContainerDied","Data":"384c034e4bb907e48b6339edcf4c102120952aa3efc955c756619a347ca0b9b5"}
Oct 29 23:08:13.871699 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.870790    4860 scope.go:122] "RemoveContainer" containerID="c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82"
Oct 29 23:08:13.871699 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.870968    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"
Oct 29 23:08:13.892987 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.892966    4860 scope.go:122] "RemoveContainer" containerID="c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82"
Oct 29 23:08:13.893383 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:08:13.893359    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82\": container with ID starting with c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82 not found: ID does not exist" containerID="c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82"
Oct 29 23:08:13.893491 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.893389    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82"} err="failed to get container status \"c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82\": rpc error: code = NotFound desc = could not find container \"c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82\": container with ID starting with c85ccdcb56a5527484469d2ce7d42b47751aad4114fc461d86a0e51b84d37c82 not found: ID does not exist"
Oct 29 23:08:13.926215 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.926180    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"]
Oct 29 23:08:13.928393 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.928368    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z794f"]
Oct 29 23:08:13.954564 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.954536    4860 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-cacerts\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:13.954564 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.954556    4860 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-dns-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:13.954564 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.954566    4860 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5567035-eebe-4ab7-8875-876f89630445-istio-kubeconfig\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:13.954828 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.954575    4860 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5567035-eebe-4ab7-8875-876f89630445-istio-csr-ca-configmap\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:13.954828 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.954584    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn8bs\" (UniqueName: \"kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-kube-api-access-kn8bs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:13.954828 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.954595    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5567035-eebe-4ab7-8875-876f89630445-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:13.954828 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:13.954604    4860 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5567035-eebe-4ab7-8875-876f89630445-local-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:14.125926 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:14.125832    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5567035-eebe-4ab7-8875-876f89630445" path="/var/lib/kubelet/pods/d5567035-eebe-4ab7-8875-876f89630445/volumes"
Oct 29 23:08:15.988835 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:15.988797    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:16.389176 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:16.389077    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:18.067341 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:08:18.067289    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:08:19.187228 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.187200    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"]
Oct 29 23:08:19.187655 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.187593    4860 cpu_manager.go:414] "containerMap: removing container" podUID="d5567035-eebe-4ab7-8875-876f89630445" containerName="discovery"
Oct 29 23:08:19.187655 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.187605    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5567035-eebe-4ab7-8875-876f89630445" containerName="discovery"
Oct 29 23:08:19.187734 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.187692    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5567035-eebe-4ab7-8875-876f89630445" containerName="discovery"
Oct 29 23:08:19.191909 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.191891    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:19.210007 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.209978    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"]
Oct 29 23:08:19.219722 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.219695    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:08:19.219855 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.219730    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:08:19.219855 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.219733    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:08:19.220726 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.220705    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:08:19.222167 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.222143    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2rk4d\" (UID: \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:19.222259 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.222216    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzgzg\" (UniqueName: \"kubernetes.io/projected/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-kube-api-access-vzgzg\") pod \"maas-api-revoke-sub-ae594979-2rk4d\" (UID: \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:19.222550 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.222530    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2rk4d\" (UID: \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:19.323985 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.323954    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2rk4d\" (UID: \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:19.324156 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.324001    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzgzg\" (UniqueName: \"kubernetes.io/projected/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-kube-api-access-vzgzg\") pod \"maas-api-revoke-sub-ae594979-2rk4d\" (UID: \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:19.324156 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.324045    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2rk4d\" (UID: \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:19.324737 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.324713    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2rk4d\" (UID: \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:19.326341 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.326301    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2rk4d\" (UID: \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:19.343761 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.343736    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzgzg\" (UniqueName: \"kubernetes.io/projected/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-kube-api-access-vzgzg\") pod \"maas-api-revoke-sub-ae594979-2rk4d\" (UID: \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:19.508099 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.508040    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:19.801805 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:08:19.801775    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcae5da_7ac5_4853_bd3d_8ca120e6a64e.slice/crio-368147effd3181cab6c807366ceba114e1cde1621db3e09be2475a4e029e6c56 WatchSource:0}: Error finding container 368147effd3181cab6c807366ceba114e1cde1621db3e09be2475a4e029e6c56: Status 404 returned error can't find the container with id 368147effd3181cab6c807366ceba114e1cde1621db3e09be2475a4e029e6c56
Oct 29 23:08:19.806480 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.803880    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"]
Oct 29 23:08:19.950151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:19.950102    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d" event={"ID":"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e","Type":"ContainerStarted","Data":"368147effd3181cab6c807366ceba114e1cde1621db3e09be2475a4e029e6c56"}
Oct 29 23:08:20.957709 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:20.957678    4860 generic.go:356] "Generic (PLEG): container finished" podID="2bcae5da-7ac5-4853-bd3d-8ca120e6a64e" containerID="029921de402b631b2c3bbd0dac042936ff508bab572b0ff8f5d6b92433dddaa0" exitCode=0
Oct 29 23:08:20.958135 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:20.957713    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d" event={"ID":"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e","Type":"ContainerDied","Data":"029921de402b631b2c3bbd0dac042936ff508bab572b0ff8f5d6b92433dddaa0"}
Oct 29 23:08:22.111817 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.111792    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:22.148436 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.148406    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-openshift-ca\") pod \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\" (UID: \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\") "
Oct 29 23:08:22.148578 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.148503    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-kube-api-access-vzgzg\" (UniqueName: \"kubernetes.io/projected/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-kube-api-access-vzgzg\") pod \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\" (UID: \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\") "
Oct 29 23:08:22.148578 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.148541    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-xks-ca\") pod \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\" (UID: \"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e\") "
Oct 29 23:08:22.148816 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.148789    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-openshift-ca" pod "2bcae5da-7ac5-4853-bd3d-8ca120e6a64e" (UID: "2bcae5da-7ac5-4853-bd3d-8ca120e6a64e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:22.150850 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.150824    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-xks-ca" pod "2bcae5da-7ac5-4853-bd3d-8ca120e6a64e" (UID: "2bcae5da-7ac5-4853-bd3d-8ca120e6a64e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:22.151806 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.151777    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-kube-api-access-vzgzg" pod "2bcae5da-7ac5-4853-bd3d-8ca120e6a64e" (UID: "2bcae5da-7ac5-4853-bd3d-8ca120e6a64e"). InnerVolumeSpecName "kube-api-access-vzgzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:22.249652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.249591    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzgzg\" (UniqueName: \"kubernetes.io/projected/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-kube-api-access-vzgzg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:22.249652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.249616    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:22.249652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.249628    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:22.969816 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.969783    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"
Oct 29 23:08:22.969999 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.969780    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d" event={"ID":"2bcae5da-7ac5-4853-bd3d-8ca120e6a64e","Type":"ContainerDied","Data":"368147effd3181cab6c807366ceba114e1cde1621db3e09be2475a4e029e6c56"}
Oct 29 23:08:22.969999 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:22.969920    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="368147effd3181cab6c807366ceba114e1cde1621db3e09be2475a4e029e6c56"
Oct 29 23:08:25.344150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:25.344119    4860 scope.go:122] "RemoveContainer" containerID="79e65189dc16243f286400ea0fe09c4259e7b91e638ae8e3ea150baf3025885d"
Oct 29 23:08:25.353218 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:25.353197    4860 scope.go:122] "RemoveContainer" containerID="5b1a882fd1a5472f33f89ac578b8eb1d8885e9be46af65b2d2e2e9373ad5f2cf"
Oct 29 23:08:33.032139 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:33.032097    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"]
Oct 29 23:08:33.046170 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:33.046137    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"]
Oct 29 23:08:33.049420 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:33.049395    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c436615-sr52r"]
Oct 29 23:08:33.050180 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:33.050145    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d69a2ad3-mt6qh"]
Oct 29 23:08:34.159202 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:34.159170    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e30c8e-fc84-4be0-8a3d-f2e01ac421e6" path="/var/lib/kubelet/pods/11e30c8e-fc84-4be0-8a3d-f2e01ac421e6/volumes"
Oct 29 23:08:34.159601 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:34.159593    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67159e2-0a49-41d4-90d1-fc3626f2c960" path="/var/lib/kubelet/pods/f67159e2-0a49-41d4-90d1-fc3626f2c960/volumes"
Oct 29 23:08:40.632892 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.632854    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"]
Oct 29 23:08:40.633287 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.633251    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2bcae5da-7ac5-4853-bd3d-8ca120e6a64e" containerName="revoke-keys"
Oct 29 23:08:40.633287 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.633264    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcae5da-7ac5-4853-bd3d-8ca120e6a64e" containerName="revoke-keys"
Oct 29 23:08:40.633398 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.633376    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcae5da-7ac5-4853-bd3d-8ca120e6a64e" containerName="revoke-keys"
Oct 29 23:08:40.637919 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.637902    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:40.654537 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.654512    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:08:40.667845 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.667823    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:08:40.667992 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.667844    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:08:40.668072 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.667995    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:08:40.671517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.671488    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"]
Oct 29 23:08:40.701703 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.701673    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s89zh\" (UniqueName: \"kubernetes.io/projected/f8819141-1618-48e6-8b48-f8ec6bfb4268-kube-api-access-s89zh\") pod \"maas-api-revoke-sub-a4aef7d7-z9xr4\" (UID: \"f8819141-1618-48e6-8b48-f8ec6bfb4268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:40.701838 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.701816    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z9xr4\" (UID: \"f8819141-1618-48e6-8b48-f8ec6bfb4268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:40.701897 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.701886    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z9xr4\" (UID: \"f8819141-1618-48e6-8b48-f8ec6bfb4268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:40.802553 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.802527    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z9xr4\" (UID: \"f8819141-1618-48e6-8b48-f8ec6bfb4268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:40.802656 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.802580    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z9xr4\" (UID: \"f8819141-1618-48e6-8b48-f8ec6bfb4268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:40.802656 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.802611    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s89zh\" (UniqueName: \"kubernetes.io/projected/f8819141-1618-48e6-8b48-f8ec6bfb4268-kube-api-access-s89zh\") pod \"maas-api-revoke-sub-a4aef7d7-z9xr4\" (UID: \"f8819141-1618-48e6-8b48-f8ec6bfb4268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:40.803156 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.803129    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z9xr4\" (UID: \"f8819141-1618-48e6-8b48-f8ec6bfb4268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:40.804864 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.804846    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z9xr4\" (UID: \"f8819141-1618-48e6-8b48-f8ec6bfb4268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:40.823091 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.823069    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s89zh\" (UniqueName: \"kubernetes.io/projected/f8819141-1618-48e6-8b48-f8ec6bfb4268-kube-api-access-s89zh\") pod \"maas-api-revoke-sub-a4aef7d7-z9xr4\" (UID: \"f8819141-1618-48e6-8b48-f8ec6bfb4268\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:40.950892 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:40.950867    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:41.250513 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:41.250164    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"]
Oct 29 23:08:42.091033 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:42.091008    4860 generic.go:356] "Generic (PLEG): container finished" podID="f8819141-1618-48e6-8b48-f8ec6bfb4268" containerID="e0f943c1ac33cb9bbcbb7c79db508b4a10586f576cdeba9ab1eda240dcf42ac3" exitCode=0
Oct 29 23:08:42.091411 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:42.091073    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4" event={"ID":"f8819141-1618-48e6-8b48-f8ec6bfb4268","Type":"ContainerDied","Data":"e0f943c1ac33cb9bbcbb7c79db508b4a10586f576cdeba9ab1eda240dcf42ac3"}
Oct 29 23:08:42.091411 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:42.091110    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4" event={"ID":"f8819141-1618-48e6-8b48-f8ec6bfb4268","Type":"ContainerStarted","Data":"d3011f9ff257fcbef119bff7cfbded6e6f8e412e59cf0f5dd8f50e02039ba755"}
Oct 29 23:08:43.240410 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:43.240382    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:43.326133 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:43.326101    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-xks-ca\") pod \"f8819141-1618-48e6-8b48-f8ec6bfb4268\" (UID: \"f8819141-1618-48e6-8b48-f8ec6bfb4268\") "
Oct 29 23:08:43.326262 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:43.326178    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8819141-1618-48e6-8b48-f8ec6bfb4268-kube-api-access-s89zh\" (UniqueName: \"kubernetes.io/projected/f8819141-1618-48e6-8b48-f8ec6bfb4268-kube-api-access-s89zh\") pod \"f8819141-1618-48e6-8b48-f8ec6bfb4268\" (UID: \"f8819141-1618-48e6-8b48-f8ec6bfb4268\") "
Oct 29 23:08:43.326262 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:43.326229    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-openshift-ca\") pod \"f8819141-1618-48e6-8b48-f8ec6bfb4268\" (UID: \"f8819141-1618-48e6-8b48-f8ec6bfb4268\") "
Oct 29 23:08:43.326623 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:43.326594    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-openshift-ca" pod "f8819141-1618-48e6-8b48-f8ec6bfb4268" (UID: "f8819141-1618-48e6-8b48-f8ec6bfb4268"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:08:43.328255 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:43.328229    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-xks-ca" pod "f8819141-1618-48e6-8b48-f8ec6bfb4268" (UID: "f8819141-1618-48e6-8b48-f8ec6bfb4268"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:08:43.328365 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:43.328314    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8819141-1618-48e6-8b48-f8ec6bfb4268-kube-api-access-s89zh" pod "f8819141-1618-48e6-8b48-f8ec6bfb4268" (UID: "f8819141-1618-48e6-8b48-f8ec6bfb4268"). InnerVolumeSpecName "kube-api-access-s89zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:08:43.427879 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:43.427824    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:43.427879 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:43.427847    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s89zh\" (UniqueName: \"kubernetes.io/projected/f8819141-1618-48e6-8b48-f8ec6bfb4268-kube-api-access-s89zh\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:43.427879 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:43.427859    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8819141-1618-48e6-8b48-f8ec6bfb4268-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:08:44.102984 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:44.102952    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"
Oct 29 23:08:44.103172 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:44.102952    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4" event={"ID":"f8819141-1618-48e6-8b48-f8ec6bfb4268","Type":"ContainerDied","Data":"d3011f9ff257fcbef119bff7cfbded6e6f8e412e59cf0f5dd8f50e02039ba755"}
Oct 29 23:08:44.103172 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:08:44.103073    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3011f9ff257fcbef119bff7cfbded6e6f8e412e59cf0f5dd8f50e02039ba755"
Oct 29 23:08:48.058691 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:08:48.058654    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:09:00.732719 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.732640    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"]
Oct 29 23:09:00.733128 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.733075    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f8819141-1618-48e6-8b48-f8ec6bfb4268" containerName="revoke-keys"
Oct 29 23:09:00.733128 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.733088    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8819141-1618-48e6-8b48-f8ec6bfb4268" containerName="revoke-keys"
Oct 29 23:09:00.733205 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.733186    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8819141-1618-48e6-8b48-f8ec6bfb4268" containerName="revoke-keys"
Oct 29 23:09:00.736417 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.736398    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:00.754633 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.754610    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"]
Oct 29 23:09:00.764643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.764620    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:09:00.764765 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.764683    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:09:00.765557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.765540    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:09:00.765657 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.765630    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:09:00.774077 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.774055    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rhsbl\" (UID: \"901d5342-6569-4d10-8f0e-3bf34fa095a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:00.774199 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.774096    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rhsbl\" (UID: \"901d5342-6569-4d10-8f0e-3bf34fa095a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:00.774272 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.774220    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w6j5\" (UniqueName: \"kubernetes.io/projected/901d5342-6569-4d10-8f0e-3bf34fa095a8-kube-api-access-2w6j5\") pod \"maas-api-revoke-sub-79805ca9-rhsbl\" (UID: \"901d5342-6569-4d10-8f0e-3bf34fa095a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:00.875510 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.875469    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rhsbl\" (UID: \"901d5342-6569-4d10-8f0e-3bf34fa095a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:00.875660 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.875544    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6j5\" (UniqueName: \"kubernetes.io/projected/901d5342-6569-4d10-8f0e-3bf34fa095a8-kube-api-access-2w6j5\") pod \"maas-api-revoke-sub-79805ca9-rhsbl\" (UID: \"901d5342-6569-4d10-8f0e-3bf34fa095a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:00.875660 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.875632    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rhsbl\" (UID: \"901d5342-6569-4d10-8f0e-3bf34fa095a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:00.876281 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.876259    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rhsbl\" (UID: \"901d5342-6569-4d10-8f0e-3bf34fa095a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:00.877907 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.877883    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rhsbl\" (UID: \"901d5342-6569-4d10-8f0e-3bf34fa095a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:00.897999 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:00.897980    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6j5\" (UniqueName: \"kubernetes.io/projected/901d5342-6569-4d10-8f0e-3bf34fa095a8-kube-api-access-2w6j5\") pod \"maas-api-revoke-sub-79805ca9-rhsbl\" (UID: \"901d5342-6569-4d10-8f0e-3bf34fa095a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:01.049391 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:01.049318    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:01.347583 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:09:01.347546    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901d5342_6569_4d10_8f0e_3bf34fa095a8.slice/crio-0266b8e59dd335b0b0e501db6ce3a860851a30f61867cd548e9429a3d953453d WatchSource:0}: Error finding container 0266b8e59dd335b0b0e501db6ce3a860851a30f61867cd548e9429a3d953453d: Status 404 returned error can't find the container with id 0266b8e59dd335b0b0e501db6ce3a860851a30f61867cd548e9429a3d953453d
Oct 29 23:09:01.348443 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:01.348420    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"]
Oct 29 23:09:02.256306 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:02.256276    4860 generic.go:356] "Generic (PLEG): container finished" podID="901d5342-6569-4d10-8f0e-3bf34fa095a8" containerID="15252cf7ef751ed5abb2cbb446bd4b6b1a8f3689211f8166c5f83f3c12dc461c" exitCode=0
Oct 29 23:09:02.256706 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:02.256371    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl" event={"ID":"901d5342-6569-4d10-8f0e-3bf34fa095a8","Type":"ContainerDied","Data":"15252cf7ef751ed5abb2cbb446bd4b6b1a8f3689211f8166c5f83f3c12dc461c"}
Oct 29 23:09:02.256706 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:02.256441    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl" event={"ID":"901d5342-6569-4d10-8f0e-3bf34fa095a8","Type":"ContainerStarted","Data":"0266b8e59dd335b0b0e501db6ce3a860851a30f61867cd548e9429a3d953453d"}
Oct 29 23:09:03.395773 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:03.395746    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:03.498502 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:03.498473    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-xks-ca\") pod \"901d5342-6569-4d10-8f0e-3bf34fa095a8\" (UID: \"901d5342-6569-4d10-8f0e-3bf34fa095a8\") "
Oct 29 23:09:03.498644 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:03.498539    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-openshift-ca\") pod \"901d5342-6569-4d10-8f0e-3bf34fa095a8\" (UID: \"901d5342-6569-4d10-8f0e-3bf34fa095a8\") "
Oct 29 23:09:03.498644 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:03.498566    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901d5342-6569-4d10-8f0e-3bf34fa095a8-kube-api-access-2w6j5\" (UniqueName: \"kubernetes.io/projected/901d5342-6569-4d10-8f0e-3bf34fa095a8-kube-api-access-2w6j5\") pod \"901d5342-6569-4d10-8f0e-3bf34fa095a8\" (UID: \"901d5342-6569-4d10-8f0e-3bf34fa095a8\") "
Oct 29 23:09:03.498892 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:03.498856    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-openshift-ca" pod "901d5342-6569-4d10-8f0e-3bf34fa095a8" (UID: "901d5342-6569-4d10-8f0e-3bf34fa095a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:03.500556 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:03.500523    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-xks-ca" pod "901d5342-6569-4d10-8f0e-3bf34fa095a8" (UID: "901d5342-6569-4d10-8f0e-3bf34fa095a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:03.500670 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:03.500642    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901d5342-6569-4d10-8f0e-3bf34fa095a8-kube-api-access-2w6j5" pod "901d5342-6569-4d10-8f0e-3bf34fa095a8" (UID: "901d5342-6569-4d10-8f0e-3bf34fa095a8"). InnerVolumeSpecName "kube-api-access-2w6j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:03.599561 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:03.599494    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:09:03.599561 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:03.599528    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901d5342-6569-4d10-8f0e-3bf34fa095a8-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:09:03.599561 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:03.599537    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w6j5\" (UniqueName: \"kubernetes.io/projected/901d5342-6569-4d10-8f0e-3bf34fa095a8-kube-api-access-2w6j5\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:09:04.268401 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:04.268366    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"
Oct 29 23:09:04.268401 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:04.268381    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl" event={"ID":"901d5342-6569-4d10-8f0e-3bf34fa095a8","Type":"ContainerDied","Data":"0266b8e59dd335b0b0e501db6ce3a860851a30f61867cd548e9429a3d953453d"}
Oct 29 23:09:04.268610 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:04.268429    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0266b8e59dd335b0b0e501db6ce3a860851a30f61867cd548e9429a3d953453d"
Oct 29 23:09:08.500524 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:08.500488    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:16.142516 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:16.142482    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:16.512067 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:16.512037    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:17.567621 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.567585    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"]
Oct 29 23:09:17.568005 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.567969    4860 cpu_manager.go:414] "containerMap: removing container" podUID="901d5342-6569-4d10-8f0e-3bf34fa095a8" containerName="revoke-keys"
Oct 29 23:09:17.568005 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.567980    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901d5342-6569-4d10-8f0e-3bf34fa095a8" containerName="revoke-keys"
Oct 29 23:09:17.568082 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.568059    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="901d5342-6569-4d10-8f0e-3bf34fa095a8" containerName="revoke-keys"
Oct 29 23:09:17.571413 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.571396    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:17.590512 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.590487    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"]
Oct 29 23:09:17.600239 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.600215    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:09:17.600400 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.600227    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:09:17.600400 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.600280    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:09:17.600509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.600401    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:09:17.659016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.658990    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2cctc\" (UID: \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:17.659124 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.659024    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h4gx\" (UniqueName: \"kubernetes.io/projected/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-kube-api-access-6h4gx\") pod \"maas-api-revoke-sub-0b7e8edb-2cctc\" (UID: \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:17.659124 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.659112    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2cctc\" (UID: \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:17.759707 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.759682    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2cctc\" (UID: \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:17.759820 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.759712    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h4gx\" (UniqueName: \"kubernetes.io/projected/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-kube-api-access-6h4gx\") pod \"maas-api-revoke-sub-0b7e8edb-2cctc\" (UID: \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:17.759820 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.759741    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2cctc\" (UID: \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:17.760429 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.760410    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2cctc\" (UID: \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:17.761996 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.761976    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2cctc\" (UID: \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:17.780152 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.780130    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h4gx\" (UniqueName: \"kubernetes.io/projected/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-kube-api-access-6h4gx\") pod \"maas-api-revoke-sub-0b7e8edb-2cctc\" (UID: \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:17.884642 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:17.884593    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:18.057368 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:09:18.057336    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:09:18.177958 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:09:18.177930    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e434bb_2ee7_498a_8cf8_b90f9f2e7ff9.slice/crio-909e73973852e9ecd4dd012888f26535e36e7e210b2ed0a1c9471485f16fdc3d WatchSource:0}: Error finding container 909e73973852e9ecd4dd012888f26535e36e7e210b2ed0a1c9471485f16fdc3d: Status 404 returned error can't find the container with id 909e73973852e9ecd4dd012888f26535e36e7e210b2ed0a1c9471485f16fdc3d
Oct 29 23:09:18.179486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:18.179390    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"]
Oct 29 23:09:18.358542 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:18.358510    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc" event={"ID":"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9","Type":"ContainerStarted","Data":"909e73973852e9ecd4dd012888f26535e36e7e210b2ed0a1c9471485f16fdc3d"}
Oct 29 23:09:18.834830 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:18.834757    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 23:09:18.858967 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:18.858946    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 23:09:19.365370 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:19.365309    4860 generic.go:356] "Generic (PLEG): container finished" podID="e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9" containerID="6bbed58a04e1b6116511ee48647be41a978a6fe3d7cba26b3d8651d4c8b66849" exitCode=0
Oct 29 23:09:19.365535 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:19.365397    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc" event={"ID":"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9","Type":"ContainerDied","Data":"6bbed58a04e1b6116511ee48647be41a978a6fe3d7cba26b3d8651d4c8b66849"}
Oct 29 23:09:20.506395 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:20.506370    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:20.685932 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:20.685907    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-kube-api-access-6h4gx\" (UniqueName: \"kubernetes.io/projected/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-kube-api-access-6h4gx\") pod \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\" (UID: \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\") "
Oct 29 23:09:20.686088 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:20.685955    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-openshift-ca\") pod \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\" (UID: \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\") "
Oct 29 23:09:20.686088 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:20.685988    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-xks-ca\") pod \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\" (UID: \"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9\") "
Oct 29 23:09:20.686320 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:20.686295    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-openshift-ca" pod "e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9" (UID: "e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:20.688107 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:20.688082    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-xks-ca" pod "e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9" (UID: "e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:20.688107 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:20.688094    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-kube-api-access-6h4gx" pod "e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9" (UID: "e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9"). InnerVolumeSpecName "kube-api-access-6h4gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:20.787577 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:20.787551    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h4gx\" (UniqueName: \"kubernetes.io/projected/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-kube-api-access-6h4gx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:09:20.787577 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:20.787574    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:09:20.787719 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:20.787587    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:09:21.377770 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:21.377742    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"
Oct 29 23:09:21.377964 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:21.377777    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc" event={"ID":"e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9","Type":"ContainerDied","Data":"909e73973852e9ecd4dd012888f26535e36e7e210b2ed0a1c9471485f16fdc3d"}
Oct 29 23:09:21.377964 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:21.377824    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="909e73973852e9ecd4dd012888f26535e36e7e210b2ed0a1c9471485f16fdc3d"
Oct 29 23:09:25.381940 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:25.381908    4860 scope.go:122] "RemoveContainer" containerID="9c6fb71d27bdd169ea9d643a39f8fb48f02e401f45b6c4e9b927d2ff96c0bed2"
Oct 29 23:09:25.393608 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:25.393534    4860 scope.go:122] "RemoveContainer" containerID="32bd5d2d44f131b1a3d910e00c99d161be9d615f485a4eb16c1ffc87fb683784"
Oct 29 23:09:25.402569 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:25.402550    4860 scope.go:122] "RemoveContainer" containerID="a7a6098d0223ba5f0e468c6605a6dee7dedc5964ed426cbe2500ca18c4bb0692"
Oct 29 23:09:25.410295 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:25.410276    4860 scope.go:122] "RemoveContainer" containerID="fe89a2f66e6ee5ce68676d4e8d03c0f7612d851390e143e7dfda52002f13fe0d"
Oct 29 23:09:25.417842 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:25.417824    4860 scope.go:122] "RemoveContainer" containerID="6ea432d0ff3d1e3e7a0eae029bf9b8b0c298b155025c58fc6b5895a04e833bc4"
Oct 29 23:09:28.298066 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.298033    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"]
Oct 29 23:09:28.298475 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.298423    4860 cpu_manager.go:414] "containerMap: removing container" podUID="e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9" containerName="revoke-keys"
Oct 29 23:09:28.298475 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.298435    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9" containerName="revoke-keys"
Oct 29 23:09:28.298553 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.298515    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9" containerName="revoke-keys"
Oct 29 23:09:28.302890 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.302866    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:28.318704 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.318680    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:09:28.318856 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.318736    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:09:28.318856 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.318760    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:09:28.319064 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.318922    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:09:28.335527 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.335498    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"]
Oct 29 23:09:28.344543 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.344515    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ntrbk\" (UID: \"274be625-bb3e-49e3-827c-36bdd3209f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:28.344679 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.344562    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ntrbk\" (UID: \"274be625-bb3e-49e3-827c-36bdd3209f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:28.344679 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.344645    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg7ww\" (UniqueName: \"kubernetes.io/projected/274be625-bb3e-49e3-827c-36bdd3209f0e-kube-api-access-lg7ww\") pod \"maas-api-revoke-sub-3d4156f1-ntrbk\" (UID: \"274be625-bb3e-49e3-827c-36bdd3209f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:28.386924 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.386897    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:28.446007 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.445979    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ntrbk\" (UID: \"274be625-bb3e-49e3-827c-36bdd3209f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:28.446131 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.446012    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ntrbk\" (UID: \"274be625-bb3e-49e3-827c-36bdd3209f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:28.446131 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.446030    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg7ww\" (UniqueName: \"kubernetes.io/projected/274be625-bb3e-49e3-827c-36bdd3209f0e-kube-api-access-lg7ww\") pod \"maas-api-revoke-sub-3d4156f1-ntrbk\" (UID: \"274be625-bb3e-49e3-827c-36bdd3209f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:28.446730 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.446708    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ntrbk\" (UID: \"274be625-bb3e-49e3-827c-36bdd3209f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:28.448286 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.448267    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ntrbk\" (UID: \"274be625-bb3e-49e3-827c-36bdd3209f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:28.467305 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.467279    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg7ww\" (UniqueName: \"kubernetes.io/projected/274be625-bb3e-49e3-827c-36bdd3209f0e-kube-api-access-lg7ww\") pod \"maas-api-revoke-sub-3d4156f1-ntrbk\" (UID: \"274be625-bb3e-49e3-827c-36bdd3209f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:28.615861 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.615814    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:28.916147 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:09:28.916116    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274be625_bb3e_49e3_827c_36bdd3209f0e.slice/crio-58bab8136afbaad3a963006160b77ca32fc62321d1c3f9ee2381a3023821137b WatchSource:0}: Error finding container 58bab8136afbaad3a963006160b77ca32fc62321d1c3f9ee2381a3023821137b: Status 404 returned error can't find the container with id 58bab8136afbaad3a963006160b77ca32fc62321d1c3f9ee2381a3023821137b
Oct 29 23:09:28.916576 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:28.916550    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"]
Oct 29 23:09:29.424453 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:29.424415    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk" event={"ID":"274be625-bb3e-49e3-827c-36bdd3209f0e","Type":"ContainerStarted","Data":"49773f3ecf9464e2023cecc530f904facf8fc3b2da6dc1d4fa0d3e378996cefe"}
Oct 29 23:09:29.424857 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:29.424471    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk" event={"ID":"274be625-bb3e-49e3-827c-36bdd3209f0e","Type":"ContainerStarted","Data":"58bab8136afbaad3a963006160b77ca32fc62321d1c3f9ee2381a3023821137b"}
Oct 29 23:09:29.467390 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:29.467320    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk" podStartSLOduration=1.467307407 podStartE2EDuration="1.467307407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:29.466667625 +0000 UTC m=+3311.491443756" watchObservedRunningTime="2026-10-29 23:09:29.467307407 +0000 UTC m=+3311.492083538"
Oct 29 23:09:30.431455 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:30.431419    4860 generic.go:356] "Generic (PLEG): container finished" podID="274be625-bb3e-49e3-827c-36bdd3209f0e" containerID="49773f3ecf9464e2023cecc530f904facf8fc3b2da6dc1d4fa0d3e378996cefe" exitCode=0
Oct 29 23:09:30.431866 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:30.431517    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk" event={"ID":"274be625-bb3e-49e3-827c-36bdd3209f0e","Type":"ContainerDied","Data":"49773f3ecf9464e2023cecc530f904facf8fc3b2da6dc1d4fa0d3e378996cefe"}
Oct 29 23:09:31.569432 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:31.569410    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:31.670779 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:31.670750    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-xks-ca\") pod \"274be625-bb3e-49e3-827c-36bdd3209f0e\" (UID: \"274be625-bb3e-49e3-827c-36bdd3209f0e\") "
Oct 29 23:09:31.670920 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:31.670802    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274be625-bb3e-49e3-827c-36bdd3209f0e-kube-api-access-lg7ww\" (UniqueName: \"kubernetes.io/projected/274be625-bb3e-49e3-827c-36bdd3209f0e-kube-api-access-lg7ww\") pod \"274be625-bb3e-49e3-827c-36bdd3209f0e\" (UID: \"274be625-bb3e-49e3-827c-36bdd3209f0e\") "
Oct 29 23:09:31.670920 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:31.670829    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-openshift-ca\") pod \"274be625-bb3e-49e3-827c-36bdd3209f0e\" (UID: \"274be625-bb3e-49e3-827c-36bdd3209f0e\") "
Oct 29 23:09:31.671265 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:31.671235    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-openshift-ca" pod "274be625-bb3e-49e3-827c-36bdd3209f0e" (UID: "274be625-bb3e-49e3-827c-36bdd3209f0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:31.672860 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:31.672833    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274be625-bb3e-49e3-827c-36bdd3209f0e-kube-api-access-lg7ww" pod "274be625-bb3e-49e3-827c-36bdd3209f0e" (UID: "274be625-bb3e-49e3-827c-36bdd3209f0e"). InnerVolumeSpecName "kube-api-access-lg7ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:31.672948 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:31.672881    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-xks-ca" pod "274be625-bb3e-49e3-827c-36bdd3209f0e" (UID: "274be625-bb3e-49e3-827c-36bdd3209f0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:31.771799 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:31.771736    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:09:31.771799 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:31.771769    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg7ww\" (UniqueName: \"kubernetes.io/projected/274be625-bb3e-49e3-827c-36bdd3209f0e-kube-api-access-lg7ww\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:09:31.771799 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:31.771780    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/274be625-bb3e-49e3-827c-36bdd3209f0e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:09:32.442998 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:32.442969    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"
Oct 29 23:09:32.443181 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:32.443004    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk" event={"ID":"274be625-bb3e-49e3-827c-36bdd3209f0e","Type":"ContainerDied","Data":"58bab8136afbaad3a963006160b77ca32fc62321d1c3f9ee2381a3023821137b"}
Oct 29 23:09:32.443181 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:32.443054    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58bab8136afbaad3a963006160b77ca32fc62321d1c3f9ee2381a3023821137b"
Oct 29 23:09:48.056422 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:09:48.056373    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:09:49.580162 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.580125    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"]
Oct 29 23:09:49.593216 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.593189    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntrbk"]
Oct 29 23:09:49.625453 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.625425    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"]
Oct 29 23:09:49.625798 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.625786    4860 cpu_manager.go:414] "containerMap: removing container" podUID="274be625-bb3e-49e3-827c-36bdd3209f0e" containerName="revoke-keys"
Oct 29 23:09:49.625846 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.625800    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274be625-bb3e-49e3-827c-36bdd3209f0e" containerName="revoke-keys"
Oct 29 23:09:49.625898 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.625890    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="274be625-bb3e-49e3-827c-36bdd3209f0e" containerName="revoke-keys"
Oct 29 23:09:49.629380 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.629363    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:49.646802 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.646778    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:09:49.646974 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.646799    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:09:49.646974 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.646858    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:09:49.647092 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.646895    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:09:49.647289 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.647270    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"]
Oct 29 23:09:49.724277 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.724244    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsbq6\" (UniqueName: \"kubernetes.io/projected/77690035-ac2d-44b2-af90-e827c1ae2d31-kube-api-access-dsbq6\") pod \"maas-api-revoke-sub-3d4156f1-75bj5\" (UID: \"77690035-ac2d-44b2-af90-e827c1ae2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:49.724451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.724303    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-75bj5\" (UID: \"77690035-ac2d-44b2-af90-e827c1ae2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:49.724451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.724426    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-75bj5\" (UID: \"77690035-ac2d-44b2-af90-e827c1ae2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:49.825483 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.825458    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-75bj5\" (UID: \"77690035-ac2d-44b2-af90-e827c1ae2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:49.825601 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.825497    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsbq6\" (UniqueName: \"kubernetes.io/projected/77690035-ac2d-44b2-af90-e827c1ae2d31-kube-api-access-dsbq6\") pod \"maas-api-revoke-sub-3d4156f1-75bj5\" (UID: \"77690035-ac2d-44b2-af90-e827c1ae2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:49.825601 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.825531    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-75bj5\" (UID: \"77690035-ac2d-44b2-af90-e827c1ae2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:49.826134 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.826107    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-75bj5\" (UID: \"77690035-ac2d-44b2-af90-e827c1ae2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:49.827869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.827847    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-75bj5\" (UID: \"77690035-ac2d-44b2-af90-e827c1ae2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:49.846114 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.846064    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsbq6\" (UniqueName: \"kubernetes.io/projected/77690035-ac2d-44b2-af90-e827c1ae2d31-kube-api-access-dsbq6\") pod \"maas-api-revoke-sub-3d4156f1-75bj5\" (UID: \"77690035-ac2d-44b2-af90-e827c1ae2d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:49.942870 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:49.942849    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:50.110311 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:50.110239    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="274be625-bb3e-49e3-827c-36bdd3209f0e" path="/var/lib/kubelet/pods/274be625-bb3e-49e3-827c-36bdd3209f0e/volumes"
Oct 29 23:09:50.237925 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:09:50.237892    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77690035_ac2d_44b2_af90_e827c1ae2d31.slice/crio-05fb11f787ca95ed5ffff1cee03213ce188c9fc8e1398cbb28da4d1e9ebb0d34 WatchSource:0}: Error finding container 05fb11f787ca95ed5ffff1cee03213ce188c9fc8e1398cbb28da4d1e9ebb0d34: Status 404 returned error can't find the container with id 05fb11f787ca95ed5ffff1cee03213ce188c9fc8e1398cbb28da4d1e9ebb0d34
Oct 29 23:09:50.239212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:50.239184    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"]
Oct 29 23:09:50.578531 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:50.578488    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5" event={"ID":"77690035-ac2d-44b2-af90-e827c1ae2d31","Type":"ContainerStarted","Data":"24d16d4bb8e69c0f7406be63582f51ab2379de20dbc911e226e2f0711ccf841c"}
Oct 29 23:09:50.578677 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:50.578551    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5" event={"ID":"77690035-ac2d-44b2-af90-e827c1ae2d31","Type":"ContainerStarted","Data":"05fb11f787ca95ed5ffff1cee03213ce188c9fc8e1398cbb28da4d1e9ebb0d34"}
Oct 29 23:09:50.620715 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:50.620675    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5" podStartSLOduration=1.6206640239999999 podStartE2EDuration="1.620664024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:50.619233339 +0000 UTC m=+3332.644009470" watchObservedRunningTime="2026-10-29 23:09:50.620664024 +0000 UTC m=+3332.645440203"
Oct 29 23:09:51.585570 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:51.585538    4860 generic.go:356] "Generic (PLEG): container finished" podID="77690035-ac2d-44b2-af90-e827c1ae2d31" containerID="24d16d4bb8e69c0f7406be63582f51ab2379de20dbc911e226e2f0711ccf841c" exitCode=0
Oct 29 23:09:51.585759 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:51.585589    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5" event={"ID":"77690035-ac2d-44b2-af90-e827c1ae2d31","Type":"ContainerDied","Data":"24d16d4bb8e69c0f7406be63582f51ab2379de20dbc911e226e2f0711ccf841c"}
Oct 29 23:09:52.726078 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:52.726057    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:52.852745 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:52.852685    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-openshift-ca\") pod \"77690035-ac2d-44b2-af90-e827c1ae2d31\" (UID: \"77690035-ac2d-44b2-af90-e827c1ae2d31\") "
Oct 29 23:09:52.852745 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:52.852732    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77690035-ac2d-44b2-af90-e827c1ae2d31-kube-api-access-dsbq6\" (UniqueName: \"kubernetes.io/projected/77690035-ac2d-44b2-af90-e827c1ae2d31-kube-api-access-dsbq6\") pod \"77690035-ac2d-44b2-af90-e827c1ae2d31\" (UID: \"77690035-ac2d-44b2-af90-e827c1ae2d31\") "
Oct 29 23:09:52.852971 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:52.852756    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-xks-ca\") pod \"77690035-ac2d-44b2-af90-e827c1ae2d31\" (UID: \"77690035-ac2d-44b2-af90-e827c1ae2d31\") "
Oct 29 23:09:52.853037 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:52.853022    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-openshift-ca" pod "77690035-ac2d-44b2-af90-e827c1ae2d31" (UID: "77690035-ac2d-44b2-af90-e827c1ae2d31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:52.854813 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:52.854787    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77690035-ac2d-44b2-af90-e827c1ae2d31-kube-api-access-dsbq6" pod "77690035-ac2d-44b2-af90-e827c1ae2d31" (UID: "77690035-ac2d-44b2-af90-e827c1ae2d31"). InnerVolumeSpecName "kube-api-access-dsbq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:52.854954 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:52.854934    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-xks-ca" pod "77690035-ac2d-44b2-af90-e827c1ae2d31" (UID: "77690035-ac2d-44b2-af90-e827c1ae2d31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:52.953369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:52.953319    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:09:52.953369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:52.953365    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsbq6\" (UniqueName: \"kubernetes.io/projected/77690035-ac2d-44b2-af90-e827c1ae2d31-kube-api-access-dsbq6\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:09:52.953509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:52.953376    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77690035-ac2d-44b2-af90-e827c1ae2d31-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:09:53.598336 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:53.598295    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"
Oct 29 23:09:53.598336 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:53.598313    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5" event={"ID":"77690035-ac2d-44b2-af90-e827c1ae2d31","Type":"ContainerDied","Data":"05fb11f787ca95ed5ffff1cee03213ce188c9fc8e1398cbb28da4d1e9ebb0d34"}
Oct 29 23:09:53.598555 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:09:53.598379    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05fb11f787ca95ed5ffff1cee03213ce188c9fc8e1398cbb28da4d1e9ebb0d34"
Oct 29 23:10:12.028285 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:12.028249    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"]
Oct 29 23:10:12.041549 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:12.041520    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbeb821-rbf8n"]
Oct 29 23:10:12.110222 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:12.110189    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee057ffa-942e-4c17-be28-ba1e52156461" path="/var/lib/kubelet/pods/ee057ffa-942e-4c17-be28-ba1e52156461/volumes"
Oct 29 23:10:13.126021 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.125984    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"]
Oct 29 23:10:13.139272 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.139242    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-75bj5"]
Oct 29 23:10:13.172977 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.172952    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"]
Oct 29 23:10:13.173352 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.173317    4860 cpu_manager.go:414] "containerMap: removing container" podUID="77690035-ac2d-44b2-af90-e827c1ae2d31" containerName="revoke-keys"
Oct 29 23:10:13.173352 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.173342    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77690035-ac2d-44b2-af90-e827c1ae2d31" containerName="revoke-keys"
Oct 29 23:10:13.173456 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.173426    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="77690035-ac2d-44b2-af90-e827c1ae2d31" containerName="revoke-keys"
Oct 29 23:10:13.176816 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.176802    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:13.192529 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.192502    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:10:13.192697 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.192536    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:10:13.192697 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.192569    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:10:13.192851 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.192638    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:10:13.196281 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.196250    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"]
Oct 29 23:10:13.324830 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.324802    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bw8bk\" (UID: \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:13.324963 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.324846    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bw8bk\" (UID: \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:13.324963 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.324876    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfncw\" (UniqueName: \"kubernetes.io/projected/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-kube-api-access-dfncw\") pod \"maas-api-revoke-sub-3d4156f1-bw8bk\" (UID: \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:13.425405 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.425341    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfncw\" (UniqueName: \"kubernetes.io/projected/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-kube-api-access-dfncw\") pod \"maas-api-revoke-sub-3d4156f1-bw8bk\" (UID: \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:13.425542 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.425409    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bw8bk\" (UID: \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:13.425542 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.425445    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bw8bk\" (UID: \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:13.426043 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.426011    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bw8bk\" (UID: \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:13.427970 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.427942    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bw8bk\" (UID: \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:13.445166 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.445135    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfncw\" (UniqueName: \"kubernetes.io/projected/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-kube-api-access-dfncw\") pod \"maas-api-revoke-sub-3d4156f1-bw8bk\" (UID: \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:13.489797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.489776    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:13.786701 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:10:13.786665    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dce7c1e_b8dd_4a2d_9c0e_a4bf8044f4d7.slice/crio-019313ef5d994c5d0d459b61c68b6b94a99b5511be75645ec3518a29c1b30c26 WatchSource:0}: Error finding container 019313ef5d994c5d0d459b61c68b6b94a99b5511be75645ec3518a29c1b30c26: Status 404 returned error can't find the container with id 019313ef5d994c5d0d459b61c68b6b94a99b5511be75645ec3518a29c1b30c26
Oct 29 23:10:13.787685 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:13.787663    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"]
Oct 29 23:10:14.111639 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:14.111602    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77690035-ac2d-44b2-af90-e827c1ae2d31" path="/var/lib/kubelet/pods/77690035-ac2d-44b2-af90-e827c1ae2d31/volumes"
Oct 29 23:10:14.710150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:14.710117    4860 generic.go:356] "Generic (PLEG): container finished" podID="6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7" containerID="cd9f48cb735f50a23ee8aa3a761bc7f576ca3ad8504e11b2d6e9e11adc837a98" exitCode=0
Oct 29 23:10:14.710502 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:14.710198    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk" event={"ID":"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7","Type":"ContainerDied","Data":"cd9f48cb735f50a23ee8aa3a761bc7f576ca3ad8504e11b2d6e9e11adc837a98"}
Oct 29 23:10:14.710502 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:14.710252    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk" event={"ID":"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7","Type":"ContainerStarted","Data":"019313ef5d994c5d0d459b61c68b6b94a99b5511be75645ec3518a29c1b30c26"}
Oct 29 23:10:15.844791 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:15.844768    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:15.947461 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:15.947432    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-kube-api-access-dfncw\" (UniqueName: \"kubernetes.io/projected/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-kube-api-access-dfncw\") pod \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\" (UID: \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\") "
Oct 29 23:10:15.947609 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:15.947501    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-xks-ca\") pod \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\" (UID: \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\") "
Oct 29 23:10:15.947660 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:15.947649    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-openshift-ca\") pod \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\" (UID: \"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7\") "
Oct 29 23:10:15.947979 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:15.947955    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-openshift-ca" pod "6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7" (UID: "6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:10:15.949575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:15.949553    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-kube-api-access-dfncw" pod "6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7" (UID: "6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7"). InnerVolumeSpecName "kube-api-access-dfncw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:10:15.949637 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:15.949572    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-xks-ca" pod "6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7" (UID: "6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:10:16.048924 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:16.048870    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:10:16.048924 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:16.048893    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfncw\" (UniqueName: \"kubernetes.io/projected/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-kube-api-access-dfncw\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:10:16.048924 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:16.048903    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:10:16.274995 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:16.274951    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:16.722320 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:16.722293    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"
Oct 29 23:10:16.722320 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:16.722290    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk" event={"ID":"6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7","Type":"ContainerDied","Data":"019313ef5d994c5d0d459b61c68b6b94a99b5511be75645ec3518a29c1b30c26"}
Oct 29 23:10:16.722544 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:16.722374    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="019313ef5d994c5d0d459b61c68b6b94a99b5511be75645ec3518a29c1b30c26"
Oct 29 23:10:18.057293 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:10:18.057251    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:10:25.455309 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:25.455275    4860 scope.go:122] "RemoveContainer" containerID="8cc1c883a0833f5110ba41d84aefadd1e2d4ff84e6b55dd53723e3a4c8867567"
Oct 29 23:10:25.464533 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:25.464511    4860 scope.go:122] "RemoveContainer" containerID="22a3a40418f047402804b93b34cda6029bff7ce97d20db6e0dd74f14c7b18147"
Oct 29 23:10:25.473047 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:25.473026    4860 scope.go:122] "RemoveContainer" containerID="6e4e1e77415cc5f166c6fa5b12757e18256cb23efb6e185412e1467e2e680b55"
Oct 29 23:10:28.265262 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.265226    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:28.660185 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.660093    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"]
Oct 29 23:10:28.660584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.660565    4860 cpu_manager.go:414] "containerMap: removing container" podUID="6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7" containerName="revoke-keys"
Oct 29 23:10:28.660711 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.660585    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7" containerName="revoke-keys"
Oct 29 23:10:28.660711 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.660662    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7" containerName="revoke-keys"
Oct 29 23:10:28.665204 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.665182    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.681572 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.681547    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-e5236b-openshift-default-dockercfg-2q75r\""
Oct 29 23:10:28.697096 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.697074    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"]
Oct 29 23:10:28.732964 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.732930    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:28.861565 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.861529    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkdg\" (UniqueName: \"kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-kube-api-access-bfkdg\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.861794 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.861587    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-data\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.861794 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.861658    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istiod-ca-cert\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.861794 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.861765    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-certs\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.861992 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.861832    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-podinfo\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.861992 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.861915    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-socket\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.861992 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.861944    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-token\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.861992 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.861974    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-credential-socket\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.862178 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.862087    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-envoy\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.962903 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.962874    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-certs\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963073 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.962930    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-podinfo\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963073 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.962974    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-socket\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963073 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.962997    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-token\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963320 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.963123    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-credential-socket\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963320 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.963193    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-envoy\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963320 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.963225    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkdg\" (UniqueName: \"kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-kube-api-access-bfkdg\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963320 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.963278    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-certs\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963320 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.963287    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-data\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963653 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.963366    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istiod-ca-cert\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963653 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.963369    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-socket\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963653 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.963488    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-credential-socket\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963653 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.963628    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-data\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.963943 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.963926    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istiod-ca-cert\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.965507 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.965484    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-envoy\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.965627 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.965609    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-podinfo\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.972594 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.972571    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:28.983963 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.983938    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-token\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:28.984523 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:28.984499    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkdg\" (UniqueName: \"kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-kube-api-access-bfkdg\") pod \"e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") " pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:29.279304 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:29.279221    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:29.589751 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:10:29.589722    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d407ac2_a99b_44a6_9f52_4feafcc58b6a.slice/crio-b23929dc441b3767bf28ebbe6be9c87290aeecdba1fe5f367a1b9cdbe99f5fc8 WatchSource:0}: Error finding container b23929dc441b3767bf28ebbe6be9c87290aeecdba1fe5f367a1b9cdbe99f5fc8: Status 404 returned error can't find the container with id b23929dc441b3767bf28ebbe6be9c87290aeecdba1fe5f367a1b9cdbe99f5fc8
Oct 29 23:10:29.590539 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:29.590514    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"]
Oct 29 23:10:29.591871 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:29.591840    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:10:29.591957 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:29.591910    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:10:29.591998 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:29.591936    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:10:29.801549 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:29.801512    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" event={"ID":"1d407ac2-a99b-44a6-9f52-4feafcc58b6a","Type":"ContainerStarted","Data":"6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5"}
Oct 29 23:10:29.801728 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:29.801571    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" event={"ID":"1d407ac2-a99b-44a6-9f52-4feafcc58b6a","Type":"ContainerStarted","Data":"b23929dc441b3767bf28ebbe6be9c87290aeecdba1fe5f367a1b9cdbe99f5fc8"}
Oct 29 23:10:29.850174 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:29.850073    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" podStartSLOduration=1.850054493 podStartE2EDuration="1.850054493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:29.84682721 +0000 UTC m=+3371.871603367" watchObservedRunningTime="2026-10-29 23:10:29.850054493 +0000 UTC m=+3371.874830624"
Oct 29 23:10:30.279953 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:30.279924    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:30.284993 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:30.284972    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:30.807558 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:30.807520    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:30.808598 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:30.808574    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:32.254570 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:32.254537    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:32.486437 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:32.486402    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:33.656878 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:33.656845    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:33.783505 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:33.783475    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:33.927891 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:33.927792    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:34.057172 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.057135    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:34.110094 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.110068    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"]
Oct 29 23:10:34.114512 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.114496    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"
Oct 29 23:10:34.129171 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.129149    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"]
Oct 29 23:10:34.139299 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.139276    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-465c3d34-dockercfg-8242k\""
Oct 29 23:10:34.139461 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.139368    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-e5236b\""
Oct 29 23:10:34.192084 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.192045    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"]
Oct 29 23:10:34.196468 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.196439    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"
Oct 29 23:10:34.207691 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.207592    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tql87\" (UniqueName: \"kubernetes.io/projected/f4bae991-2eef-4d17-aae8-f677eca7a8b5-kube-api-access-tql87\") pod \"payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg\" (UID: \"f4bae991-2eef-4d17-aae8-f677eca7a8b5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"
Oct 29 23:10:34.207691 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.207676    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bae991-2eef-4d17-aae8-f677eca7a8b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg\" (UID: \"f4bae991-2eef-4d17-aae8-f677eca7a8b5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"
Oct 29 23:10:34.207903 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.207719    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k4kr\" (UniqueName: \"kubernetes.io/projected/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-kube-api-access-8k4kr\") pod \"payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf\" (UID: \"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"
Oct 29 23:10:34.207903 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.207744    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf\" (UID: \"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"
Oct 29 23:10:34.214584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.214558    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"]
Oct 29 23:10:34.308350 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.308299    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k4kr\" (UniqueName: \"kubernetes.io/projected/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-kube-api-access-8k4kr\") pod \"payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf\" (UID: \"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"
Oct 29 23:10:34.308484 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.308365    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf\" (UID: \"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"
Oct 29 23:10:34.308484 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.308445    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tql87\" (UniqueName: \"kubernetes.io/projected/f4bae991-2eef-4d17-aae8-f677eca7a8b5-kube-api-access-tql87\") pod \"payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg\" (UID: \"f4bae991-2eef-4d17-aae8-f677eca7a8b5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"
Oct 29 23:10:34.308572 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.308528    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bae991-2eef-4d17-aae8-f677eca7a8b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg\" (UID: \"f4bae991-2eef-4d17-aae8-f677eca7a8b5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"
Oct 29 23:10:34.309115 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.309096    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf\" (UID: \"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"
Oct 29 23:10:34.309232 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.309214    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bae991-2eef-4d17-aae8-f677eca7a8b5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg\" (UID: \"f4bae991-2eef-4d17-aae8-f677eca7a8b5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"
Oct 29 23:10:34.328610 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.328588    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tql87\" (UniqueName: \"kubernetes.io/projected/f4bae991-2eef-4d17-aae8-f677eca7a8b5-kube-api-access-tql87\") pod \"payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg\" (UID: \"f4bae991-2eef-4d17-aae8-f677eca7a8b5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"
Oct 29 23:10:34.328740 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.328718    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k4kr\" (UniqueName: \"kubernetes.io/projected/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-kube-api-access-8k4kr\") pod \"payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf\" (UID: \"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"
Oct 29 23:10:34.428486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.428455    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"
Oct 29 23:10:34.515934 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.515903    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"
Oct 29 23:10:34.727517 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:10:34.727482    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb5ee4f_d770_4f0d_bef7_1ad123bc558f.slice/crio-f141ede87edf01e667ca7d8aeab0d848dd7c3c62bfc6754b0a72f5a9e54559ef WatchSource:0}: Error finding container f141ede87edf01e667ca7d8aeab0d848dd7c3c62bfc6754b0a72f5a9e54559ef: Status 404 returned error can't find the container with id f141ede87edf01e667ca7d8aeab0d848dd7c3c62bfc6754b0a72f5a9e54559ef
Oct 29 23:10:34.728803 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.728734    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"]
Oct 29 23:10:34.811299 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:10:34.811268    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bae991_2eef_4d17_aae8_f677eca7a8b5.slice/crio-971fba95a56500b49c769ff5bbc56e74d7e63d1d99f1b56fa3172ab424e22d89 WatchSource:0}: Error finding container 971fba95a56500b49c769ff5bbc56e74d7e63d1d99f1b56fa3172ab424e22d89: Status 404 returned error can't find the container with id 971fba95a56500b49c769ff5bbc56e74d7e63d1d99f1b56fa3172ab424e22d89
Oct 29 23:10:34.813675 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.813645    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"]
Oct 29 23:10:34.834200 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.834168    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf" event={"ID":"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f","Type":"ContainerStarted","Data":"5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5"}
Oct 29 23:10:34.834336 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.834239    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf" event={"ID":"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f","Type":"ContainerStarted","Data":"f141ede87edf01e667ca7d8aeab0d848dd7c3c62bfc6754b0a72f5a9e54559ef"}
Oct 29 23:10:34.834336 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.834311    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"
Oct 29 23:10:34.836060 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.836034    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg" event={"ID":"f4bae991-2eef-4d17-aae8-f677eca7a8b5","Type":"ContainerStarted","Data":"971fba95a56500b49c769ff5bbc56e74d7e63d1d99f1b56fa3172ab424e22d89"}
Oct 29 23:10:34.877776 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:34.877733    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf" podStartSLOduration=0.877720453 podStartE2EDuration="877.720453ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:34.87513059 +0000 UTC m=+3376.899906721" watchObservedRunningTime="2026-10-29 23:10:34.877720453 +0000 UTC m=+3376.902496584"
Oct 29 23:10:35.490017 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:35.489968    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:35.843294 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:35.843198    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg" event={"ID":"f4bae991-2eef-4d17-aae8-f677eca7a8b5","Type":"ContainerStarted","Data":"99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465"}
Oct 29 23:10:35.843735 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:35.843372    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"
Oct 29 23:10:35.884826 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:35.884778    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg" podStartSLOduration=1.8847653439999998 podStartE2EDuration="1.884765344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:35.882544029 +0000 UTC m=+3377.907320189" watchObservedRunningTime="2026-10-29 23:10:35.884765344 +0000 UTC m=+3377.909541514"
Oct 29 23:10:37.084178 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:37.084138    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:38.748935 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:38.748900    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:40.318573 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:40.318531    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:41.766302 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:41.766267    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:43.295018 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:43.294921    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:44.875600 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:44.875564    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:45.846612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:45.846581    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"
Oct 29 23:10:46.058750 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:46.058359    4860 patch_prober.go:29] interesting pod/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": read tcp 10.132.0.2:34500->10.132.1.127:15021: read: connection reset by peer" start-of-body=
Oct 29 23:10:46.058750 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:46.058425    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": read tcp 10.132.0.2:34500->10.132.1.127:15021: read: connection reset by peer"
Oct 29 23:10:46.794193 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:46.794150    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:46.851644 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:46.851615    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"
Oct 29 23:10:46.932889 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:46.932855    4860 generic.go:356] "Generic (PLEG): container finished" podID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerID="6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5" exitCode=0
Oct 29 23:10:46.933052 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:46.932923    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" event={"ID":"1d407ac2-a99b-44a6-9f52-4feafcc58b6a","Type":"ContainerDied","Data":"6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5"}
Oct 29 23:10:46.933318 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:46.933294    4860 scope.go:122] "RemoveContainer" containerID="6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5"
Oct 29 23:10:46.934479 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:46.934452    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:10:46.934549 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:46.934521    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:10:46.934592 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:46.934553    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:10:47.225848 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:47.225815    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:47.940461 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:47.940428    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" event={"ID":"1d407ac2-a99b-44a6-9f52-4feafcc58b6a","Type":"ContainerStarted","Data":"bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea"}
Oct 29 23:10:47.959946 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:47.959922    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:48.062402 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:10:48.062364    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:10:48.096825 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:48.096761    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:48.284409 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:48.280682    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:48.284409 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:48.280805    4860 patch_prober.go:29] interesting pod/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body=
Oct 29 23:10:48.284409 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:48.280862    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused"
Oct 29 23:10:48.519712 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:48.519672    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:49.289349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:49.286729    4860 patch_prober.go:29] interesting pod/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body=
Oct 29 23:10:49.289349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:49.286807    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused"
Oct 29 23:10:50.287349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.286662    4860 patch_prober.go:29] interesting pod/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body=
Oct 29 23:10:50.287349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.286737    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused"
Oct 29 23:10:50.346096 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.346065    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:50.513502 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.513465    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"]
Oct 29 23:10:50.516359 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.516312    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.548065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.547981    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e5236b\"/\"default-dockercfg-txwwt\""
Oct 29 23:10:50.548065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.547981    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e5236b\"/\"openshift-service-ca.crt\""
Oct 29 23:10:50.548065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.548018    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e5236b\"/\"kube-root-ca.crt\""
Oct 29 23:10:50.548065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.548049    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e5236b\"/\"simulator-single-main-e5236b-kserve-self-signed-certs\""
Oct 29 23:10:50.551884 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.551854    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"]
Oct 29 23:10:50.608457 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.608421    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b762fea-81bc-4310-9d86-6e64283742a0-tls-certs\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.608639 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.608496    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26cjq\" (UniqueName: \"kubernetes.io/projected/5b762fea-81bc-4310-9d86-6e64283742a0-kube-api-access-26cjq\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.608639 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.608588    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-tmp-dir\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.608785 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.608666    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-dshm\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.608785 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.608747    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-home\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.608906 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.608845    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-model-cache\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.609149 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.609128    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:50.709878 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.709845    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b762fea-81bc-4310-9d86-6e64283742a0-tls-certs\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.709878 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.709879    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26cjq\" (UniqueName: \"kubernetes.io/projected/5b762fea-81bc-4310-9d86-6e64283742a0-kube-api-access-26cjq\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.710124 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.709910    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-tmp-dir\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.710124 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.709933    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-dshm\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.710124 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.709987    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-home\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.710124 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.710032    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-model-cache\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.710412 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.710398    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-tmp-dir\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.710478 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.710430    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-home\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.710527 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.710475    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-model-cache\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.712345 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.712310    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-dshm\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.712619 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.712601    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b762fea-81bc-4310-9d86-6e64283742a0-tls-certs\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.729794 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.729773    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26cjq\" (UniqueName: \"kubernetes.io/projected/5b762fea-81bc-4310-9d86-6e64283742a0-kube-api-access-26cjq\") pod \"simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") " pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:50.830081 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:50.829988    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:51.153191 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:10:51.153159    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b762fea_81bc_4310_9d86_6e64283742a0.slice/crio-9ec92b4f0eadc5b36e5b691b571b52d636379baa457147ed3955907c3f96e73a WatchSource:0}: Error finding container 9ec92b4f0eadc5b36e5b691b571b52d636379baa457147ed3955907c3f96e73a: Status 404 returned error can't find the container with id 9ec92b4f0eadc5b36e5b691b571b52d636379baa457147ed3955907c3f96e73a
Oct 29 23:10:51.153815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:51.153789    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"]
Oct 29 23:10:51.291347 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:51.287689    4860 patch_prober.go:29] interesting pod/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body=
Oct 29 23:10:51.291347 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:51.287743    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused"
Oct 29 23:10:51.901594 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:51.901558    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:51.994240 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:51.994200    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p" event={"ID":"5b762fea-81bc-4310-9d86-6e64283742a0","Type":"ContainerStarted","Data":"bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba"}
Oct 29 23:10:51.994461 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:51.994255    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p" event={"ID":"5b762fea-81bc-4310-9d86-6e64283742a0","Type":"ContainerStarted","Data":"9ec92b4f0eadc5b36e5b691b571b52d636379baa457147ed3955907c3f96e73a"}
Oct 29 23:10:52.035986 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:52.035937    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p" podStartSLOduration=2.035923206 podStartE2EDuration="2.035923206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:52.033957177 +0000 UTC m=+3394.058733328" watchObservedRunningTime="2026-10-29 23:10:52.035923206 +0000 UTC m=+3394.060699336"
Oct 29 23:10:52.288349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:52.285761    4860 patch_prober.go:29] interesting pod/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body=
Oct 29 23:10:52.288349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:52.285822    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused"
Oct 29 23:10:52.830225 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:52.830174    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:10:53.512360 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:53.512294    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:53.629096 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:53.629054    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:53.629539 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:53.629520    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:53.630482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:53.630462    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:10:55.194510 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:55.194475    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:56.705939 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:56.705904    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:58.395570 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:58.395525    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:59.875315 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:10:59.875284    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:00.830431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:00.830394    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:11:00.842635 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:00.842608    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:11:01.059223 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:01.059192    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:11:01.366155 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:01.366120    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:02.984944 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:02.984912    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:04.497939 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:04.497900    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:06.188398 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:06.188364    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:07.756496 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:07.756460    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:09.355215 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:09.355182    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:10.945633 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:10.945596    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:12.472713 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:12.472682    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:12.776674 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:12.776597    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"]
Oct 29 23:11:12.781529 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:12.781512    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:12.797951 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:12.797926    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"]
Oct 29 23:11:12.808225 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:12.808199    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:11:12.808366 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:12.808227    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:11:12.808366 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:12.808287    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:11:12.810075 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:12.810055    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:11:12.909841 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:12.909806    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03fc0ef3-v2qf2\" (UID: \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:12.910002 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:12.909855    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03fc0ef3-v2qf2\" (UID: \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:12.910002 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:12.909986    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7967j\" (UniqueName: \"kubernetes.io/projected/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-kube-api-access-7967j\") pod \"maas-api-revoke-sub-03fc0ef3-v2qf2\" (UID: \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:13.011113 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:13.011074    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03fc0ef3-v2qf2\" (UID: \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:13.011306 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:13.011124    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03fc0ef3-v2qf2\" (UID: \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:13.011306 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:13.011165    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7967j\" (UniqueName: \"kubernetes.io/projected/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-kube-api-access-7967j\") pod \"maas-api-revoke-sub-03fc0ef3-v2qf2\" (UID: \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:13.011872 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:13.011845    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03fc0ef3-v2qf2\" (UID: \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:13.013559 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:13.013536    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03fc0ef3-v2qf2\" (UID: \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:13.031904 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:13.031854    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7967j\" (UniqueName: \"kubernetes.io/projected/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-kube-api-access-7967j\") pod \"maas-api-revoke-sub-03fc0ef3-v2qf2\" (UID: \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:13.095182 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:13.095154    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:13.392227 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:11:13.392186    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdcf4b0_cbd3_4dab_9875_8955142c5a43.slice/crio-9a515be1da1ffdabceb9094f0b35ca4ed2112f79be8c0790d21b3ab05f52a34f WatchSource:0}: Error finding container 9a515be1da1ffdabceb9094f0b35ca4ed2112f79be8c0790d21b3ab05f52a34f: Status 404 returned error can't find the container with id 9a515be1da1ffdabceb9094f0b35ca4ed2112f79be8c0790d21b3ab05f52a34f
Oct 29 23:11:13.394265 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:13.394235    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"]
Oct 29 23:11:14.111700 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:14.111664    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:14.123123 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:14.123092    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2" event={"ID":"7cdcf4b0-cbd3-4dab-9875-8955142c5a43","Type":"ContainerStarted","Data":"87b5f24286f66f9d63f45fbebd1410d997f6d18f79107bcfcdf03d4e19dd225e"}
Oct 29 23:11:14.123253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:14.123148    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2" event={"ID":"7cdcf4b0-cbd3-4dab-9875-8955142c5a43","Type":"ContainerStarted","Data":"9a515be1da1ffdabceb9094f0b35ca4ed2112f79be8c0790d21b3ab05f52a34f"}
Oct 29 23:11:14.163866 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:14.163816    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2" podStartSLOduration=2.163799107 podStartE2EDuration="2.163799107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:14.162340008 +0000 UTC m=+3416.187116149" watchObservedRunningTime="2026-10-29 23:11:14.163799107 +0000 UTC m=+3416.188575241"
Oct 29 23:11:15.130252 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:15.130221    4860 generic.go:356] "Generic (PLEG): container finished" podID="7cdcf4b0-cbd3-4dab-9875-8955142c5a43" containerID="87b5f24286f66f9d63f45fbebd1410d997f6d18f79107bcfcdf03d4e19dd225e" exitCode=0
Oct 29 23:11:15.130664 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:15.130308    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2" event={"ID":"7cdcf4b0-cbd3-4dab-9875-8955142c5a43","Type":"ContainerDied","Data":"87b5f24286f66f9d63f45fbebd1410d997f6d18f79107bcfcdf03d4e19dd225e"}
Oct 29 23:11:15.354689 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:15.354655    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:15.696192 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:15.696156    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:16.279375 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:16.279350    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:16.377546 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:16.377516    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:16.439646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:16.439618    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-openshift-ca\") pod \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\" (UID: \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\") "
Oct 29 23:11:16.439790 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:16.439670    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-xks-ca\") pod \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\" (UID: \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\") "
Oct 29 23:11:16.439790 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:16.439744    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-kube-api-access-7967j\" (UniqueName: \"kubernetes.io/projected/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-kube-api-access-7967j\") pod \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\" (UID: \"7cdcf4b0-cbd3-4dab-9875-8955142c5a43\") "
Oct 29 23:11:16.439995 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:16.439972    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-openshift-ca" pod "7cdcf4b0-cbd3-4dab-9875-8955142c5a43" (UID: "7cdcf4b0-cbd3-4dab-9875-8955142c5a43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:16.440074 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:16.440053    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:16.441862 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:16.441840    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-kube-api-access-7967j" pod "7cdcf4b0-cbd3-4dab-9875-8955142c5a43" (UID: "7cdcf4b0-cbd3-4dab-9875-8955142c5a43"). InnerVolumeSpecName "kube-api-access-7967j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:16.441960 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:16.441902    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-xks-ca" pod "7cdcf4b0-cbd3-4dab-9875-8955142c5a43" (UID: "7cdcf4b0-cbd3-4dab-9875-8955142c5a43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:16.541071 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:16.541047    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:16.541071 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:16.541072    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7967j\" (UniqueName: \"kubernetes.io/projected/7cdcf4b0-cbd3-4dab-9875-8955142c5a43-kube-api-access-7967j\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:17.145465 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:17.145427    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2" event={"ID":"7cdcf4b0-cbd3-4dab-9875-8955142c5a43","Type":"ContainerDied","Data":"9a515be1da1ffdabceb9094f0b35ca4ed2112f79be8c0790d21b3ab05f52a34f"}
Oct 29 23:11:17.145676 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:17.145491    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a515be1da1ffdabceb9094f0b35ca4ed2112f79be8c0790d21b3ab05f52a34f"
Oct 29 23:11:17.145676 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:17.145438    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"
Oct 29 23:11:17.348336 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:17.348283    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:18.057339 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:11:18.057289    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:11:18.890770 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:18.890738    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:20.568150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:20.568114    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:22.098483 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:22.098451    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:23.807468 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:23.807426    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:23.920785 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:23.920729    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:24.011980 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.011945    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"]
Oct 29 23:11:24.012403 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.012389    4860 cpu_manager.go:414] "containerMap: removing container" podUID="7cdcf4b0-cbd3-4dab-9875-8955142c5a43" containerName="revoke-keys"
Oct 29 23:11:24.012458 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.012405    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cdcf4b0-cbd3-4dab-9875-8955142c5a43" containerName="revoke-keys"
Oct 29 23:11:24.012509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.012498    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cdcf4b0-cbd3-4dab-9875-8955142c5a43" containerName="revoke-keys"
Oct 29 23:11:24.015612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.015597    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:24.033640 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.033616    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"]
Oct 29 23:11:24.043408 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.043385    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:11:24.043530 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.043421    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:11:24.043530 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.043447    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:11:24.044514 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.044496    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:11:24.208155 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.208126    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd\" (UID: \"37f6408a-df17-4587-b5e5-39079d4543f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:24.208356 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.208175    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd\" (UID: \"37f6408a-df17-4587-b5e5-39079d4543f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:24.208440 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.208350    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwdmz\" (UniqueName: \"kubernetes.io/projected/37f6408a-df17-4587-b5e5-39079d4543f2-kube-api-access-nwdmz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd\" (UID: \"37f6408a-df17-4587-b5e5-39079d4543f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:24.309086 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.309059    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd\" (UID: \"37f6408a-df17-4587-b5e5-39079d4543f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:24.309221 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.309098    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd\" (UID: \"37f6408a-df17-4587-b5e5-39079d4543f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:24.309221 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.309172    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwdmz\" (UniqueName: \"kubernetes.io/projected/37f6408a-df17-4587-b5e5-39079d4543f2-kube-api-access-nwdmz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd\" (UID: \"37f6408a-df17-4587-b5e5-39079d4543f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:24.309732 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.309706    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd\" (UID: \"37f6408a-df17-4587-b5e5-39079d4543f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:24.311627 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.311600    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd\" (UID: \"37f6408a-df17-4587-b5e5-39079d4543f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:24.329048 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.329024    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwdmz\" (UniqueName: \"kubernetes.io/projected/37f6408a-df17-4587-b5e5-39079d4543f2-kube-api-access-nwdmz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd\" (UID: \"37f6408a-df17-4587-b5e5-39079d4543f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:24.628947 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.628872    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:24.928653 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:11:24.928617    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f6408a_df17_4587_b5e5_39079d4543f2.slice/crio-e4139732647159fbd4b749f95492ee7ed77319986b2b23a11d5593c641deb7de WatchSource:0}: Error finding container e4139732647159fbd4b749f95492ee7ed77319986b2b23a11d5593c641deb7de: Status 404 returned error can't find the container with id e4139732647159fbd4b749f95492ee7ed77319986b2b23a11d5593c641deb7de
Oct 29 23:11:24.930106 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:24.930083    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"]
Oct 29 23:11:25.192416 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:25.192377    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd" event={"ID":"37f6408a-df17-4587-b5e5-39079d4543f2","Type":"ContainerStarted","Data":"1098b32770d4212c284a7f80efee806a1995496b183e66434bdc3eabdf25f488"}
Oct 29 23:11:25.192535 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:25.192477    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd" event={"ID":"37f6408a-df17-4587-b5e5-39079d4543f2","Type":"ContainerStarted","Data":"e4139732647159fbd4b749f95492ee7ed77319986b2b23a11d5593c641deb7de"}
Oct 29 23:11:25.234687 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:25.234647    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd" podStartSLOduration=2.234633512 podStartE2EDuration="2.234633512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:25.231997515 +0000 UTC m=+3427.256773647" watchObservedRunningTime="2026-10-29 23:11:25.234633512 +0000 UTC m=+3427.259409642"
Oct 29 23:11:25.501975 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:25.501943    4860 scope.go:122] "RemoveContainer" containerID="67aeabac6abcaa676e90ff82bc59e9976ff734d124595c01b30942bbea24bce7"
Oct 29 23:11:25.703797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:25.703755    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:26.199641 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:26.199609    4860 generic.go:356] "Generic (PLEG): container finished" podID="37f6408a-df17-4587-b5e5-39079d4543f2" containerID="1098b32770d4212c284a7f80efee806a1995496b183e66434bdc3eabdf25f488" exitCode=0
Oct 29 23:11:26.200022 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:26.199652    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd" event={"ID":"37f6408a-df17-4587-b5e5-39079d4543f2","Type":"ContainerDied","Data":"1098b32770d4212c284a7f80efee806a1995496b183e66434bdc3eabdf25f488"}
Oct 29 23:11:27.337596 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:27.337571    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:27.382907 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:27.382860    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:27.435643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:27.435610    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f6408a-df17-4587-b5e5-39079d4543f2-kube-api-access-nwdmz\" (UniqueName: \"kubernetes.io/projected/37f6408a-df17-4587-b5e5-39079d4543f2-kube-api-access-nwdmz\") pod \"37f6408a-df17-4587-b5e5-39079d4543f2\" (UID: \"37f6408a-df17-4587-b5e5-39079d4543f2\") "
Oct 29 23:11:27.435842 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:27.435702    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-xks-ca\") pod \"37f6408a-df17-4587-b5e5-39079d4543f2\" (UID: \"37f6408a-df17-4587-b5e5-39079d4543f2\") "
Oct 29 23:11:27.435842 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:27.435736    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-openshift-ca\") pod \"37f6408a-df17-4587-b5e5-39079d4543f2\" (UID: \"37f6408a-df17-4587-b5e5-39079d4543f2\") "
Oct 29 23:11:27.436158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:27.436127    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-openshift-ca" pod "37f6408a-df17-4587-b5e5-39079d4543f2" (UID: "37f6408a-df17-4587-b5e5-39079d4543f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:27.437804 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:27.437782    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f6408a-df17-4587-b5e5-39079d4543f2-kube-api-access-nwdmz" pod "37f6408a-df17-4587-b5e5-39079d4543f2" (UID: "37f6408a-df17-4587-b5e5-39079d4543f2"). InnerVolumeSpecName "kube-api-access-nwdmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:27.437950 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:27.437929    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-xks-ca" pod "37f6408a-df17-4587-b5e5-39079d4543f2" (UID: "37f6408a-df17-4587-b5e5-39079d4543f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:27.536835 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:27.536782    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwdmz\" (UniqueName: \"kubernetes.io/projected/37f6408a-df17-4587-b5e5-39079d4543f2-kube-api-access-nwdmz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:27.536835 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:27.536807    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:27.536835 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:27.536818    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f6408a-df17-4587-b5e5-39079d4543f2-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:28.211800 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:28.211772    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"
Oct 29 23:11:28.211964 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:28.211809    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd" event={"ID":"37f6408a-df17-4587-b5e5-39079d4543f2","Type":"ContainerDied","Data":"e4139732647159fbd4b749f95492ee7ed77319986b2b23a11d5593c641deb7de"}
Oct 29 23:11:28.211964 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:28.211857    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4139732647159fbd4b749f95492ee7ed77319986b2b23a11d5593c641deb7de"
Oct 29 23:11:28.803860 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:28.803824    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"]
Oct 29 23:11:28.888025 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:28.887995    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:29.095661 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:29.095577    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:30.591682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:30.591648    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:32.209143 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:32.209078    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:33.707363 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:33.707309    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:34.460614 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:34.460582    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"]
Oct 29 23:11:34.496066 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:34.496038    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"]
Oct 29 23:11:34.629403 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:34.629373    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:34.746840 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:34.746750    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:35.109942 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:35.109861    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:37.322361 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:37.322307    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:37.437990 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:37.437953    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"]
Oct 29 23:11:37.450858 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:37.450829    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e5236b-jpznd"]
Oct 29 23:11:37.456839 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:37.456812    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:38.004292 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:38.004254    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:38.110021 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:38.109992    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f6408a-df17-4587-b5e5-39079d4543f2" path="/var/lib/kubelet/pods/37f6408a-df17-4587-b5e5-39079d4543f2/volumes"
Oct 29 23:11:39.461342 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:39.461288    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg" podUID="f4bae991-2eef-4d17-aae8-f677eca7a8b5" containerName="payload-processing" containerID="cri-o://99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465" gracePeriod=25
Oct 29 23:11:39.496517 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:39.496463    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf" podUID="4eb5ee4f-d770-4f0d-bef7-1ad123bc558f" containerName="payload-pre-processing" containerID="cri-o://5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5" gracePeriod=25
Oct 29 23:11:43.824042 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:43.824004    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p" podUID="5b762fea-81bc-4310-9d86-6e64283742a0" containerName="main" containerID="cri-o://bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba" gracePeriod=45
Oct 29 23:11:44.072696 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.072670    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:11:44.092801 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.092736    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:44.186355 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.186299    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-home\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-home\") pod \"5b762fea-81bc-4310-9d86-6e64283742a0\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") "
Oct 29 23:11:44.186355 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.186355    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-dshm\") pod \"5b762fea-81bc-4310-9d86-6e64283742a0\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") "
Oct 29 23:11:44.186682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.186446    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b762fea-81bc-4310-9d86-6e64283742a0-kube-api-access-26cjq\" (UniqueName: \"kubernetes.io/projected/5b762fea-81bc-4310-9d86-6e64283742a0-kube-api-access-26cjq\") pod \"5b762fea-81bc-4310-9d86-6e64283742a0\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") "
Oct 29 23:11:44.186682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.186465    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-model-cache\") pod \"5b762fea-81bc-4310-9d86-6e64283742a0\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") "
Oct 29 23:11:44.186682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.186489    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-tmp-dir\") pod \"5b762fea-81bc-4310-9d86-6e64283742a0\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") "
Oct 29 23:11:44.186682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.186535    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b762fea-81bc-4310-9d86-6e64283742a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b762fea-81bc-4310-9d86-6e64283742a0-tls-certs\") pod \"5b762fea-81bc-4310-9d86-6e64283742a0\" (UID: \"5b762fea-81bc-4310-9d86-6e64283742a0\") "
Oct 29 23:11:44.186682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.186566    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-home" pod "5b762fea-81bc-4310-9d86-6e64283742a0" (UID: "5b762fea-81bc-4310-9d86-6e64283742a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:44.187038 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.186767    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-model-cache" pod "5b762fea-81bc-4310-9d86-6e64283742a0" (UID: "5b762fea-81bc-4310-9d86-6e64283742a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:44.187038 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.186776    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-tmp-dir" pod "5b762fea-81bc-4310-9d86-6e64283742a0" (UID: "5b762fea-81bc-4310-9d86-6e64283742a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:44.187038 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.186947    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:44.187038 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.186967    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:44.187038 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.186983    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:44.188874 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.188837    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b762fea-81bc-4310-9d86-6e64283742a0-kube-api-access-26cjq" pod "5b762fea-81bc-4310-9d86-6e64283742a0" (UID: "5b762fea-81bc-4310-9d86-6e64283742a0"). InnerVolumeSpecName "kube-api-access-26cjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:44.189004 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.188981    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b762fea-81bc-4310-9d86-6e64283742a0-tls-certs" pod "5b762fea-81bc-4310-9d86-6e64283742a0" (UID: "5b762fea-81bc-4310-9d86-6e64283742a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:44.189056 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.188989    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-dshm" pod "5b762fea-81bc-4310-9d86-6e64283742a0" (UID: "5b762fea-81bc-4310-9d86-6e64283742a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:44.220394 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.220362    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:44.287858 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.287825    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b762fea-81bc-4310-9d86-6e64283742a0-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:44.287858 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.287854    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26cjq\" (UniqueName: \"kubernetes.io/projected/5b762fea-81bc-4310-9d86-6e64283742a0-kube-api-access-26cjq\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:44.287858 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.287867    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b762fea-81bc-4310-9d86-6e64283742a0-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:44.319526 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.319496    4860 generic.go:356] "Generic (PLEG): container finished" podID="5b762fea-81bc-4310-9d86-6e64283742a0" containerID="bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba" exitCode=0
Oct 29 23:11:44.319651 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.319556    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"
Oct 29 23:11:44.319651 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.319592    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p" event={"ID":"5b762fea-81bc-4310-9d86-6e64283742a0","Type":"ContainerDied","Data":"bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba"}
Oct 29 23:11:44.319651 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.319637    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p" event={"ID":"5b762fea-81bc-4310-9d86-6e64283742a0","Type":"ContainerDied","Data":"9ec92b4f0eadc5b36e5b691b571b52d636379baa457147ed3955907c3f96e73a"}
Oct 29 23:11:44.319761 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.319656    4860 scope.go:122] "RemoveContainer" containerID="bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba"
Oct 29 23:11:44.329853 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.329834    4860 scope.go:122] "RemoveContainer" containerID="bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba"
Oct 29 23:11:44.330119 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:11:44.330094    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba\": container with ID starting with bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba not found: ID does not exist" containerID="bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba"
Oct 29 23:11:44.330220 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.330124    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba"} err="failed to get container status \"bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba\": rpc error: code = NotFound desc = could not find container \"bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba\": container with ID starting with bf341defee8b5f44ecc5963440ba9f04e37c891b2fe110326ed1fef5bf3c92ba not found: ID does not exist"
Oct 29 23:11:44.395988 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.395912    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"]
Oct 29 23:11:44.411431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.411402    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p"]
Oct 29 23:11:44.460833 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.460808    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"]
Oct 29 23:11:44.461071 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.461047    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy" containerID="cri-o://bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea" gracePeriod=30
Oct 29 23:11:44.518490 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:44.518464    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:45.046311 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:45.046267    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-e5236b/simulator-single-main-e5236b-kserve-6f8b6944c5-qtk4p" podUID="5b762fea-81bc-4310-9d86-6e64283742a0" containerName="main" probeResult="failure" output="Get \"https://10.132.1.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:11:45.687303 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:45.687266    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"]
Oct 29 23:11:45.700886 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:45.700857    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw8bk"]
Oct 29 23:11:45.845501 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:45.845466    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf" podUID="4eb5ee4f-d770-4f0d-bef7-1ad123bc558f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:11:46.110749 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:46.110673    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b762fea-81bc-4310-9d86-6e64283742a0" path="/var/lib/kubelet/pods/5b762fea-81bc-4310-9d86-6e64283742a0/volumes"
Oct 29 23:11:46.111120 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:46.111088    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7" path="/var/lib/kubelet/pods/6dce7c1e-b8dd-4a2d-9c0e-a4bf8044f4d7/volumes"
Oct 29 23:11:46.850198 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:46.850163    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg" podUID="f4bae991-2eef-4d17-aae8-f677eca7a8b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:11:48.056286 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:11:48.056244    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:11:49.712805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.712783    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:11:49.837781 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.837717    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-kube-api-access-bfkdg\" (UniqueName: \"kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-kube-api-access-bfkdg\") pod \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") "
Oct 29 23:11:49.837781 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.837779    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-data\") pod \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") "
Oct 29 23:11:49.837993 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.837822    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istiod-ca-cert\") pod \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") "
Oct 29 23:11:49.837993 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.837857    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-envoy\") pod \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") "
Oct 29 23:11:49.837993 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.837909    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-token\" (UniqueName: \"kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-token\") pod \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") "
Oct 29 23:11:49.837993 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.837942    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-socket\") pod \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") "
Oct 29 23:11:49.837993 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.837981    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-certs\") pod \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") "
Oct 29 23:11:49.838292 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.838015    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-podinfo\") pod \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") "
Oct 29 23:11:49.838292 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.838041    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-credential-socket\") pod \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\" (UID: \"1d407ac2-a99b-44a6-9f52-4feafcc58b6a\") "
Oct 29 23:11:49.838292 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.838139    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-data" pod "1d407ac2-a99b-44a6-9f52-4feafcc58b6a" (UID: "1d407ac2-a99b-44a6-9f52-4feafcc58b6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:49.838511 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.838363    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-socket" pod "1d407ac2-a99b-44a6-9f52-4feafcc58b6a" (UID: "1d407ac2-a99b-44a6-9f52-4feafcc58b6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:49.838511 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.838397    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istiod-ca-cert" pod "1d407ac2-a99b-44a6-9f52-4feafcc58b6a" (UID: "1d407ac2-a99b-44a6-9f52-4feafcc58b6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:49.838511 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.838414    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:49.838685 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.838556    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-credential-socket" pod "1d407ac2-a99b-44a6-9f52-4feafcc58b6a" (UID: "1d407ac2-a99b-44a6-9f52-4feafcc58b6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:49.838685 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.838595    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-certs" pod "1d407ac2-a99b-44a6-9f52-4feafcc58b6a" (UID: "1d407ac2-a99b-44a6-9f52-4feafcc58b6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:49.840098 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.840068    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-kube-api-access-bfkdg" pod "1d407ac2-a99b-44a6-9f52-4feafcc58b6a" (UID: "1d407ac2-a99b-44a6-9f52-4feafcc58b6a"). InnerVolumeSpecName "kube-api-access-bfkdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:49.840229 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.840210    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-podinfo" pod "1d407ac2-a99b-44a6-9f52-4feafcc58b6a" (UID: "1d407ac2-a99b-44a6-9f52-4feafcc58b6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:11:49.840300 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.840260    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-token" pod "1d407ac2-a99b-44a6-9f52-4feafcc58b6a" (UID: "1d407ac2-a99b-44a6-9f52-4feafcc58b6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:49.840483 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.840466    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-envoy" pod "1d407ac2-a99b-44a6-9f52-4feafcc58b6a" (UID: "1d407ac2-a99b-44a6-9f52-4feafcc58b6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:11:49.939191 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.939162    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:49.939191 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.939187    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:49.939383 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.939196    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:49.939383 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.939206    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfkdg\" (UniqueName: \"kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-kube-api-access-bfkdg\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:49.939383 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.939216    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:49.939383 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.939224    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:49.939383 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.939231    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:49.939383 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:49.939238    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d407ac2-a99b-44a6-9f52-4feafcc58b6a-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:11:50.364809 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:50.364773    4860 generic.go:356] "Generic (PLEG): container finished" podID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerID="bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea" exitCode=0
Oct 29 23:11:50.364963 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:50.364822    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" event={"ID":"1d407ac2-a99b-44a6-9f52-4feafcc58b6a","Type":"ContainerDied","Data":"bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea"}
Oct 29 23:11:50.364963 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:50.364841    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"
Oct 29 23:11:50.364963 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:50.364863    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z" event={"ID":"1d407ac2-a99b-44a6-9f52-4feafcc58b6a","Type":"ContainerDied","Data":"b23929dc441b3767bf28ebbe6be9c87290aeecdba1fe5f367a1b9cdbe99f5fc8"}
Oct 29 23:11:50.364963 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:50.364885    4860 scope.go:122] "RemoveContainer" containerID="bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea"
Oct 29 23:11:50.374524 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:50.374506    4860 scope.go:122] "RemoveContainer" containerID="6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5"
Oct 29 23:11:50.383569 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:50.383547    4860 scope.go:122] "RemoveContainer" containerID="bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea"
Oct 29 23:11:50.383871 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:11:50.383852    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea\": container with ID starting with bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea not found: ID does not exist" containerID="bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea"
Oct 29 23:11:50.383920 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:50.383881    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea"} err="failed to get container status \"bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea\": rpc error: code = NotFound desc = could not find container \"bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea\": container with ID starting with bb3aca02ca0c6011c688223cc9dd8bfc16b5104c945ffb10625d527ff2653bea not found: ID does not exist"
Oct 29 23:11:50.383920 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:50.383901    4860 scope.go:122] "RemoveContainer" containerID="6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5"
Oct 29 23:11:50.384181 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:11:50.384154    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5\": container with ID starting with 6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5 not found: ID does not exist" containerID="6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5"
Oct 29 23:11:50.384249 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:50.384191    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5"} err="failed to get container status \"6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5\": rpc error: code = NotFound desc = could not find container \"6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5\": container with ID starting with 6d163391fefdfe0feee81eb416799615c3de089d0750c9ea86009ba1fa60cba5 not found: ID does not exist"
Oct 29 23:11:50.433161 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:50.433127    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"]
Oct 29 23:11:50.446981 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:50.446956    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e5236b-openshift-default-547784c65-qg68z"]
Oct 29 23:11:51.076141 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:51.076095    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:51.202008 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:51.201981    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:52.110256 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:52.110223    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" path="/var/lib/kubelet/pods/1d407ac2-a99b-44a6-9f52-4feafcc58b6a/volumes"
Oct 29 23:11:55.845223 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:55.845184    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf" podUID="4eb5ee4f-d770-4f0d-bef7-1ad123bc558f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:11:56.850232 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:56.850198    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg" podUID="f4bae991-2eef-4d17-aae8-f677eca7a8b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:11:59.747644 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:59.747616    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"
Oct 29 23:11:59.771815 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:59.771794    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"
Oct 29 23:11:59.923343 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:59.923298    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-plugins-config-volume\") pod \"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f\" (UID: \"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f\") "
Oct 29 23:11:59.923511 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:59.923363    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4bae991-2eef-4d17-aae8-f677eca7a8b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bae991-2eef-4d17-aae8-f677eca7a8b5-plugins-config-volume\") pod \"f4bae991-2eef-4d17-aae8-f677eca7a8b5\" (UID: \"f4bae991-2eef-4d17-aae8-f677eca7a8b5\") "
Oct 29 23:11:59.923511 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:59.923446    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4bae991-2eef-4d17-aae8-f677eca7a8b5-kube-api-access-tql87\" (UniqueName: \"kubernetes.io/projected/f4bae991-2eef-4d17-aae8-f677eca7a8b5-kube-api-access-tql87\") pod \"f4bae991-2eef-4d17-aae8-f677eca7a8b5\" (UID: \"f4bae991-2eef-4d17-aae8-f677eca7a8b5\") "
Oct 29 23:11:59.923511 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:59.923499    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-kube-api-access-8k4kr\" (UniqueName: \"kubernetes.io/projected/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-kube-api-access-8k4kr\") pod \"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f\" (UID: \"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f\") "
Oct 29 23:11:59.923756 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:59.923730    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-plugins-config-volume" pod "4eb5ee4f-d770-4f0d-bef7-1ad123bc558f" (UID: "4eb5ee4f-d770-4f0d-bef7-1ad123bc558f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:59.923810 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:59.923784    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4bae991-2eef-4d17-aae8-f677eca7a8b5-plugins-config-volume" pod "f4bae991-2eef-4d17-aae8-f677eca7a8b5" (UID: "f4bae991-2eef-4d17-aae8-f677eca7a8b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:59.925556 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:59.925528    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4bae991-2eef-4d17-aae8-f677eca7a8b5-kube-api-access-tql87" pod "f4bae991-2eef-4d17-aae8-f677eca7a8b5" (UID: "f4bae991-2eef-4d17-aae8-f677eca7a8b5"). InnerVolumeSpecName "kube-api-access-tql87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:59.925652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:11:59.925590    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-kube-api-access-8k4kr" pod "4eb5ee4f-d770-4f0d-bef7-1ad123bc558f" (UID: "4eb5ee4f-d770-4f0d-bef7-1ad123bc558f"). InnerVolumeSpecName "kube-api-access-8k4kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:00.025147 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.025086    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tql87\" (UniqueName: \"kubernetes.io/projected/f4bae991-2eef-4d17-aae8-f677eca7a8b5-kube-api-access-tql87\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:12:00.025147 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.025109    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k4kr\" (UniqueName: \"kubernetes.io/projected/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-kube-api-access-8k4kr\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:12:00.025147 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.025120    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:12:00.025147 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.025130    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4bae991-2eef-4d17-aae8-f677eca7a8b5-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:12:00.423038 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.422950    4860 generic.go:356] "Generic (PLEG): container finished" podID="4eb5ee4f-d770-4f0d-bef7-1ad123bc558f" containerID="5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5" exitCode=0
Oct 29 23:12:00.423038 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.423002    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"
Oct 29 23:12:00.423261 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.423037    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf" event={"ID":"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f","Type":"ContainerDied","Data":"5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5"}
Oct 29 23:12:00.423261 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.423103    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf" event={"ID":"4eb5ee4f-d770-4f0d-bef7-1ad123bc558f","Type":"ContainerDied","Data":"f141ede87edf01e667ca7d8aeab0d848dd7c3c62bfc6754b0a72f5a9e54559ef"}
Oct 29 23:12:00.423261 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.423129    4860 scope.go:122] "RemoveContainer" containerID="5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5"
Oct 29 23:12:00.424540 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.424518    4860 generic.go:356] "Generic (PLEG): container finished" podID="f4bae991-2eef-4d17-aae8-f677eca7a8b5" containerID="99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465" exitCode=0
Oct 29 23:12:00.424690 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.424567    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"
Oct 29 23:12:00.424690 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.424609    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg" event={"ID":"f4bae991-2eef-4d17-aae8-f677eca7a8b5","Type":"ContainerDied","Data":"99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465"}
Oct 29 23:12:00.424690 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.424651    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg" event={"ID":"f4bae991-2eef-4d17-aae8-f677eca7a8b5","Type":"ContainerDied","Data":"971fba95a56500b49c769ff5bbc56e74d7e63d1d99f1b56fa3172ab424e22d89"}
Oct 29 23:12:00.432894 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.432869    4860 scope.go:122] "RemoveContainer" containerID="5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5"
Oct 29 23:12:00.433180 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:12:00.433154    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5\": container with ID starting with 5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5 not found: ID does not exist" containerID="5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5"
Oct 29 23:12:00.433285 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.433187    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5"} err="failed to get container status \"5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5\": rpc error: code = NotFound desc = could not find container \"5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5\": container with ID starting with 5f4e5da94ce455029fe15034afd47fc07b47efbbb6174bd086d8e72ee9db29b5 not found: ID does not exist"
Oct 29 23:12:00.433285 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.433204    4860 scope.go:122] "RemoveContainer" containerID="99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465"
Oct 29 23:12:00.441570 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.441546    4860 scope.go:122] "RemoveContainer" containerID="99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465"
Oct 29 23:12:00.441835 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:12:00.441817    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465\": container with ID starting with 99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465 not found: ID does not exist" containerID="99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465"
Oct 29 23:12:00.441887 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.441841    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465"} err="failed to get container status \"99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465\": rpc error: code = NotFound desc = could not find container \"99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465\": container with ID starting with 99afba6847faa131e9eeecd1534cced9cb20332b4b2e65e5b58e6a5d9b1cd465 not found: ID does not exist"
Oct 29 23:12:00.476856 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.476811    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"]
Oct 29 23:12:00.489837 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.489811    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e5236b-fcf66d4f44xjqg"]
Oct 29 23:12:00.513376 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.513355    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"]
Oct 29 23:12:00.517287 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:00.517262    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e5236b-7548fnkkbf"]
Oct 29 23:12:02.110173 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:02.110142    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb5ee4f-d770-4f0d-bef7-1ad123bc558f" path="/var/lib/kubelet/pods/4eb5ee4f-d770-4f0d-bef7-1ad123bc558f/volumes"
Oct 29 23:12:02.110615 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:02.110532    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4bae991-2eef-4d17-aae8-f677eca7a8b5" path="/var/lib/kubelet/pods/f4bae991-2eef-4d17-aae8-f677eca7a8b5/volumes"
Oct 29 23:12:05.627901 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:05.627870    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:06.076112 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:06.076076    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:12.097797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:12.097764    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:16.508198 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:16.508163    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:18.054264 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:12:18.054229    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:12:48.055864 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:12:48.055830    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:12:53.826581 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:53.826538    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:58.962577 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:58.962544    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:59.076480 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:12:59.076447    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:16.652255 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:16.652221    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:17.227285 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:17.227255    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:18.055251 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:13:18.055215    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:13:23.029042 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:23.029007    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"]
Oct 29 23:13:23.041052 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:23.041022    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rk4d"]
Oct 29 23:13:24.109647 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:24.109616    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bcae5da-7ac5-4853-bd3d-8ca120e6a64e" path="/var/lib/kubelet/pods/2bcae5da-7ac5-4853-bd3d-8ca120e6a64e/volumes"
Oct 29 23:13:28.863502 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863472    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"]
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863833    4860 cpu_manager.go:414] "containerMap: removing container" podUID="5b762fea-81bc-4310-9d86-6e64283742a0" containerName="main"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863844    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b762fea-81bc-4310-9d86-6e64283742a0" containerName="main"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863856    4860 cpu_manager.go:414] "containerMap: removing container" podUID="4eb5ee4f-d770-4f0d-bef7-1ad123bc558f" containerName="payload-pre-processing"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863861    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb5ee4f-d770-4f0d-bef7-1ad123bc558f" containerName="payload-pre-processing"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863879    4860 cpu_manager.go:414] "containerMap: removing container" podUID="37f6408a-df17-4587-b5e5-39079d4543f2" containerName="revoke-keys"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863884    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f6408a-df17-4587-b5e5-39079d4543f2" containerName="revoke-keys"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863892    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863896    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863914    4860 cpu_manager.go:414] "containerMap: removing container" podUID="f4bae991-2eef-4d17-aae8-f677eca7a8b5" containerName="payload-processing"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863919    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4bae991-2eef-4d17-aae8-f677eca7a8b5" containerName="payload-processing"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863935    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.863939    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.864013    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f6408a-df17-4587-b5e5-39079d4543f2" containerName="revoke-keys"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.864021    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.864030    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4bae991-2eef-4d17-aae8-f677eca7a8b5" containerName="payload-processing"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.864038    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d407ac2-a99b-44a6-9f52-4feafcc58b6a" containerName="istio-proxy"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.864046    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b762fea-81bc-4310-9d86-6e64283742a0" containerName="main"
Oct 29 23:13:28.864158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.864051    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb5ee4f-d770-4f0d-bef7-1ad123bc558f" containerName="payload-pre-processing"
Oct 29 23:13:28.867248 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.867231    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:28.884188 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.884159    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:13:28.897513 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.897488    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:13:28.897638 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.897573    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:13:28.897638 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.897597    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:13:28.900096 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.900075    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"]
Oct 29 23:13:28.962061 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.962038    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-7cmrf\" (UID: \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:28.962205 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.962095    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j849h\" (UniqueName: \"kubernetes.io/projected/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-kube-api-access-j849h\") pod \"maas-api-revoke-sub-d576e68b-7cmrf\" (UID: \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:28.962277 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:28.962247    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-7cmrf\" (UID: \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:29.063132 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:29.063100    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-7cmrf\" (UID: \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:29.063256 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:29.063162    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j849h\" (UniqueName: \"kubernetes.io/projected/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-kube-api-access-j849h\") pod \"maas-api-revoke-sub-d576e68b-7cmrf\" (UID: \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:29.063256 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:29.063236    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-7cmrf\" (UID: \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:29.063921 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:29.063901    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-7cmrf\" (UID: \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:29.065591 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:29.065573    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-7cmrf\" (UID: \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:29.085788 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:29.085764    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j849h\" (UniqueName: \"kubernetes.io/projected/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-kube-api-access-j849h\") pod \"maas-api-revoke-sub-d576e68b-7cmrf\" (UID: \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:29.180432 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:29.180413    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:29.477306 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:13:29.477275    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2810b57f_2f9a_4e4b_bc6e_0291c3f203de.slice/crio-f10be12ca712eb96acddc13895209f32e490f50355dafce495957dbdac7782ea WatchSource:0}: Error finding container f10be12ca712eb96acddc13895209f32e490f50355dafce495957dbdac7782ea: Status 404 returned error can't find the container with id f10be12ca712eb96acddc13895209f32e490f50355dafce495957dbdac7782ea
Oct 29 23:13:29.478077 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:29.478052    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"]
Oct 29 23:13:29.935373 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:29.935316    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf" event={"ID":"2810b57f-2f9a-4e4b-bc6e-0291c3f203de","Type":"ContainerStarted","Data":"4de8151ad9865c62bdc78608f6c5715f21ed3083c2244e76dac38b32afba3983"}
Oct 29 23:13:29.935731 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:29.935386    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf" event={"ID":"2810b57f-2f9a-4e4b-bc6e-0291c3f203de","Type":"ContainerStarted","Data":"f10be12ca712eb96acddc13895209f32e490f50355dafce495957dbdac7782ea"}
Oct 29 23:13:29.976689 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:29.976641    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf" podStartSLOduration=1.976628118 podStartE2EDuration="1.976628118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:29.975877064 +0000 UTC m=+3552.000653218" watchObservedRunningTime="2026-10-29 23:13:29.976628118 +0000 UTC m=+3552.001404650"
Oct 29 23:13:30.942400 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:30.942362    4860 generic.go:356] "Generic (PLEG): container finished" podID="2810b57f-2f9a-4e4b-bc6e-0291c3f203de" containerID="4de8151ad9865c62bdc78608f6c5715f21ed3083c2244e76dac38b32afba3983" exitCode=0
Oct 29 23:13:30.942802 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:30.942446    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf" event={"ID":"2810b57f-2f9a-4e4b-bc6e-0291c3f203de","Type":"ContainerDied","Data":"4de8151ad9865c62bdc78608f6c5715f21ed3083c2244e76dac38b32afba3983"}
Oct 29 23:13:32.083017 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.082990    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:32.187784 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.187756    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-kube-api-access-j849h\" (UniqueName: \"kubernetes.io/projected/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-kube-api-access-j849h\") pod \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\" (UID: \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\") "
Oct 29 23:13:32.187944 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.187825    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-xks-ca\") pod \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\" (UID: \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\") "
Oct 29 23:13:32.187994 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.187947    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-openshift-ca\") pod \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\" (UID: \"2810b57f-2f9a-4e4b-bc6e-0291c3f203de\") "
Oct 29 23:13:32.188370 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.188342    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-openshift-ca" pod "2810b57f-2f9a-4e4b-bc6e-0291c3f203de" (UID: "2810b57f-2f9a-4e4b-bc6e-0291c3f203de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:13:32.189946 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.189922    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-kube-api-access-j849h" pod "2810b57f-2f9a-4e4b-bc6e-0291c3f203de" (UID: "2810b57f-2f9a-4e4b-bc6e-0291c3f203de"). InnerVolumeSpecName "kube-api-access-j849h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:13:32.189946 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.189931    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-xks-ca" pod "2810b57f-2f9a-4e4b-bc6e-0291c3f203de" (UID: "2810b57f-2f9a-4e4b-bc6e-0291c3f203de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:13:32.289391 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.289313    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:13:32.289391 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.289356    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:13:32.289391 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.289370    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j849h\" (UniqueName: \"kubernetes.io/projected/2810b57f-2f9a-4e4b-bc6e-0291c3f203de-kube-api-access-j849h\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:13:32.955271 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.955232    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf" event={"ID":"2810b57f-2f9a-4e4b-bc6e-0291c3f203de","Type":"ContainerDied","Data":"f10be12ca712eb96acddc13895209f32e490f50355dafce495957dbdac7782ea"}
Oct 29 23:13:32.955271 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.955264    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"
Oct 29 23:13:32.955501 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:32.955292    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f10be12ca712eb96acddc13895209f32e490f50355dafce495957dbdac7782ea"
Oct 29 23:13:43.787186 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.787149    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"]
Oct 29 23:13:43.787802 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.787718    4860 cpu_manager.go:414] "containerMap: removing container" podUID="2810b57f-2f9a-4e4b-bc6e-0291c3f203de" containerName="revoke-keys"
Oct 29 23:13:43.787802 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.787739    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2810b57f-2f9a-4e4b-bc6e-0291c3f203de" containerName="revoke-keys"
Oct 29 23:13:43.787955 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.787869    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="2810b57f-2f9a-4e4b-bc6e-0291c3f203de" containerName="revoke-keys"
Oct 29 23:13:43.791761 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.791736    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:43.835350 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.813135    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"]
Oct 29 23:13:43.839068 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.837681    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:13:43.839285 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.837774    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:13:43.839438 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.837819    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:13:43.839552 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.837908    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:13:43.934386 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.925875    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6hg6t\" (UID: \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:43.934386 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.926007    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9nqz\" (UniqueName: \"kubernetes.io/projected/963dfe0e-ca6f-4421-88a3-6a92352c17d4-kube-api-access-j9nqz\") pod \"maas-api-revoke-sub-d2bc93bc-6hg6t\" (UID: \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:43.934386 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:43.926051    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6hg6t\" (UID: \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:44.042995 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:44.042907    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6hg6t\" (UID: \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:44.049904 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:44.047106    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9nqz\" (UniqueName: \"kubernetes.io/projected/963dfe0e-ca6f-4421-88a3-6a92352c17d4-kube-api-access-j9nqz\") pod \"maas-api-revoke-sub-d2bc93bc-6hg6t\" (UID: \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:44.049904 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:44.047162    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6hg6t\" (UID: \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:44.049904 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:44.044124    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6hg6t\" (UID: \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:44.050809 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:44.050786    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6hg6t\" (UID: \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:44.050968 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:44.050834    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"]
Oct 29 23:13:44.054580 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:44.054559    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z9xr4"]
Oct 29 23:13:44.090340 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:44.085485    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9nqz\" (UniqueName: \"kubernetes.io/projected/963dfe0e-ca6f-4421-88a3-6a92352c17d4-kube-api-access-j9nqz\") pod \"maas-api-revoke-sub-d2bc93bc-6hg6t\" (UID: \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:44.142346 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:44.130057    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8819141-1618-48e6-8b48-f8ec6bfb4268" path="/var/lib/kubelet/pods/f8819141-1618-48e6-8b48-f8ec6bfb4268/volumes"
Oct 29 23:13:44.160434 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:44.160416    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:44.643705 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:44.640393    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"]
Oct 29 23:13:45.061419 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:45.061382    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t" event={"ID":"963dfe0e-ca6f-4421-88a3-6a92352c17d4","Type":"ContainerStarted","Data":"3e7db1fcc139d24a3a47a9a7d260adeedb10872c5efd20eec368e9fecf59c183"}
Oct 29 23:13:46.070665 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:46.070585    4860 generic.go:356] "Generic (PLEG): container finished" podID="963dfe0e-ca6f-4421-88a3-6a92352c17d4" containerID="404c5dd05ebee27186b3dc493d310533072003985fca1fe17b063819d41ff7d3" exitCode=0
Oct 29 23:13:46.071102 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:46.070669    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t" event={"ID":"963dfe0e-ca6f-4421-88a3-6a92352c17d4","Type":"ContainerDied","Data":"404c5dd05ebee27186b3dc493d310533072003985fca1fe17b063819d41ff7d3"}
Oct 29 23:13:46.407550 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:46.407471    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:47.211013 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:47.210991    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:47.378569 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:47.378508    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-xks-ca\") pod \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\" (UID: \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\") "
Oct 29 23:13:47.378731 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:47.378580    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963dfe0e-ca6f-4421-88a3-6a92352c17d4-kube-api-access-j9nqz\" (UniqueName: \"kubernetes.io/projected/963dfe0e-ca6f-4421-88a3-6a92352c17d4-kube-api-access-j9nqz\") pod \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\" (UID: \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\") "
Oct 29 23:13:47.378731 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:47.378613    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-openshift-ca\") pod \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\" (UID: \"963dfe0e-ca6f-4421-88a3-6a92352c17d4\") "
Oct 29 23:13:47.379010 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:47.378990    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-openshift-ca" pod "963dfe0e-ca6f-4421-88a3-6a92352c17d4" (UID: "963dfe0e-ca6f-4421-88a3-6a92352c17d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:13:47.380573 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:47.380546    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-xks-ca" pod "963dfe0e-ca6f-4421-88a3-6a92352c17d4" (UID: "963dfe0e-ca6f-4421-88a3-6a92352c17d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:13:47.380963 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:47.380937    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963dfe0e-ca6f-4421-88a3-6a92352c17d4-kube-api-access-j9nqz" pod "963dfe0e-ca6f-4421-88a3-6a92352c17d4" (UID: "963dfe0e-ca6f-4421-88a3-6a92352c17d4"). InnerVolumeSpecName "kube-api-access-j9nqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:13:47.480166 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:47.480129    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:13:47.480166 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:47.480161    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9nqz\" (UniqueName: \"kubernetes.io/projected/963dfe0e-ca6f-4421-88a3-6a92352c17d4-kube-api-access-j9nqz\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:13:47.480415 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:47.480175    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963dfe0e-ca6f-4421-88a3-6a92352c17d4-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:13:48.055433 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:13:48.055395    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:13:48.082846 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:48.082824    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"
Oct 29 23:13:48.083004 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:48.082865    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t" event={"ID":"963dfe0e-ca6f-4421-88a3-6a92352c17d4","Type":"ContainerDied","Data":"3e7db1fcc139d24a3a47a9a7d260adeedb10872c5efd20eec368e9fecf59c183"}
Oct 29 23:13:48.083004 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:48.082931    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e7db1fcc139d24a3a47a9a7d260adeedb10872c5efd20eec368e9fecf59c183"
Oct 29 23:13:55.509241 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:55.509199    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:55.920405 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:55.920303    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"]
Oct 29 23:13:55.920741 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:55.920724    4860 cpu_manager.go:414] "containerMap: removing container" podUID="963dfe0e-ca6f-4421-88a3-6a92352c17d4" containerName="revoke-keys"
Oct 29 23:13:55.920741 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:55.920739    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963dfe0e-ca6f-4421-88a3-6a92352c17d4" containerName="revoke-keys"
Oct 29 23:13:55.920900 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:55.920804    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="963dfe0e-ca6f-4421-88a3-6a92352c17d4" containerName="revoke-keys"
Oct 29 23:13:55.924008 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:55.923992    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:55.940983 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:55.940958    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-063169-openshift-default-dockercfg-zwd9x\""
Oct 29 23:13:55.956719 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:55.956697    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"]
Oct 29 23:13:56.010467 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.010440    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:56.054830 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.054795    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-certs\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.055031 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.054846    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-credential-socket\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.055031 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.054974    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-token\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.055150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.055048    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-envoy\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.055150 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.055075    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-podinfo\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.055259 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.055185    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istiod-ca-cert\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.055314 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.055277    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-data\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.055405 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.055381    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krbrj\" (UniqueName: \"kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-kube-api-access-krbrj\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.055467 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.055422    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-socket\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.156910 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.156880    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-certs\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.156910 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.156911    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-credential-socket\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.157119 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.157032    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-token\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.157119 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.157096    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-envoy\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.157226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.157124    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-podinfo\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.157226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.157151    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istiod-ca-cert\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.157226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.157202    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-data\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.157437 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.157260    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbrj\" (UniqueName: \"kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-kube-api-access-krbrj\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.157437 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.157301    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-socket\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.157437 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.157309    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-certs\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.157437 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.157364    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-credential-socket\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.157679 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.157615    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-socket\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.157720 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.157697    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-data\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.157941 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.157921    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istiod-ca-cert\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.159596 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.159569    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-envoy\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.159753 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.159732    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-podinfo\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.177503 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.177453    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-token\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.177869 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.177850    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbrj\" (UniqueName: \"kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-kube-api-access-krbrj\") pod \"e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") " pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.230841 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.230811    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:56.236832 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.236810    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:56.552768 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:13:56.552739    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc668e357_23fd_44b9_8aa3_e3aae2f061c8.slice/crio-7cf04a0ef5540f0c0c93ce823ec751787229bbc133532574130b27da99a29cfe WatchSource:0}: Error finding container 7cf04a0ef5540f0c0c93ce823ec751787229bbc133532574130b27da99a29cfe: Status 404 returned error can't find the container with id 7cf04a0ef5540f0c0c93ce823ec751787229bbc133532574130b27da99a29cfe
Oct 29 23:13:56.554451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.554422    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"]
Oct 29 23:13:56.554850 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.554817    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:13:56.554916 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.554890    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:13:56.554953 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:56.554921    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:13:57.134595 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:57.134556    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" event={"ID":"c668e357-23fd-44b9-8aa3-e3aae2f061c8","Type":"ContainerStarted","Data":"ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33"}
Oct 29 23:13:57.134774 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:57.134609    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" event={"ID":"c668e357-23fd-44b9-8aa3-e3aae2f061c8","Type":"ContainerStarted","Data":"7cf04a0ef5540f0c0c93ce823ec751787229bbc133532574130b27da99a29cfe"}
Oct 29 23:13:57.214018 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:57.213969    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" podStartSLOduration=2.213956334 podStartE2EDuration="2.213956334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:57.199666057 +0000 UTC m=+3579.224442200" watchObservedRunningTime="2026-10-29 23:13:57.213956334 +0000 UTC m=+3579.238732465"
Oct 29 23:13:57.237080 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:57.237053    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:57.242074 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:57.242050    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:58.139585 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:58.139551    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:58.140619 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:58.140592    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:13:59.664505 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:59.664472    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:59.828573 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:13:59.828542    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:01.161342 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.161301    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:01.289159 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.289124    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:01.417907 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.417780    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:01.490866 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.490833    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"]
Oct 29 23:14:01.494708 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.494684    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"
Oct 29 23:14:01.506218 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.506193    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdr4\" (UniqueName: \"kubernetes.io/projected/3b5c0e17-151f-4471-b12c-d7b7228bf80a-kube-api-access-lbdr4\") pod \"payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg\" (UID: \"3b5c0e17-151f-4471-b12c-d7b7228bf80a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"
Oct 29 23:14:01.506314 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.506257    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5c0e17-151f-4471-b12c-d7b7228bf80a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg\" (UID: \"3b5c0e17-151f-4471-b12c-d7b7228bf80a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"
Oct 29 23:14:01.513527 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.513503    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"]
Oct 29 23:14:01.523050 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.523030    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-063169\""
Oct 29 23:14:01.523156 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.523133    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-85d4e7ba-dockercfg-nvlmf\""
Oct 29 23:14:01.607302 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.607263    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5c0e17-151f-4471-b12c-d7b7228bf80a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg\" (UID: \"3b5c0e17-151f-4471-b12c-d7b7228bf80a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"
Oct 29 23:14:01.607492 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.607417    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdr4\" (UniqueName: \"kubernetes.io/projected/3b5c0e17-151f-4471-b12c-d7b7228bf80a-kube-api-access-lbdr4\") pod \"payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg\" (UID: \"3b5c0e17-151f-4471-b12c-d7b7228bf80a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"
Oct 29 23:14:01.607834 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.607810    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"]
Oct 29 23:14:01.608678 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.608648    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5c0e17-151f-4471-b12c-d7b7228bf80a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg\" (UID: \"3b5c0e17-151f-4471-b12c-d7b7228bf80a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"
Oct 29 23:14:01.611785 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.611769    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"
Oct 29 23:14:01.633026 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.633002    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"]
Oct 29 23:14:01.633134 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.633117    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdr4\" (UniqueName: \"kubernetes.io/projected/3b5c0e17-151f-4471-b12c-d7b7228bf80a-kube-api-access-lbdr4\") pod \"payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg\" (UID: \"3b5c0e17-151f-4471-b12c-d7b7228bf80a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"
Oct 29 23:14:01.708594 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.708564    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkr7\" (UniqueName: \"kubernetes.io/projected/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-kube-api-access-gzkr7\") pod \"payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw\" (UID: \"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"
Oct 29 23:14:01.708763 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.708603    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw\" (UID: \"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"
Oct 29 23:14:01.808477 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.808443    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"
Oct 29 23:14:01.809899 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.809866    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkr7\" (UniqueName: \"kubernetes.io/projected/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-kube-api-access-gzkr7\") pod \"payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw\" (UID: \"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"
Oct 29 23:14:01.810031 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.809911    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw\" (UID: \"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"
Oct 29 23:14:01.810693 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.810673    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw\" (UID: \"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"
Oct 29 23:14:01.832916 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.832888    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkr7\" (UniqueName: \"kubernetes.io/projected/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-kube-api-access-gzkr7\") pod \"payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw\" (UID: \"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"
Oct 29 23:14:01.924200 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:01.924166    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"
Oct 29 23:14:02.106518 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:14:02.106486    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5c0e17_151f_4471_b12c_d7b7228bf80a.slice/crio-299a638611226af73240ee46ec74495ad576b2b46ecd015ea55cca1a8d09883d WatchSource:0}: Error finding container 299a638611226af73240ee46ec74495ad576b2b46ecd015ea55cca1a8d09883d: Status 404 returned error can't find the container with id 299a638611226af73240ee46ec74495ad576b2b46ecd015ea55cca1a8d09883d
Oct 29 23:14:02.111610 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:02.111584    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"]
Oct 29 23:14:02.165547 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:02.165520    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg" event={"ID":"3b5c0e17-151f-4471-b12c-d7b7228bf80a","Type":"ContainerStarted","Data":"299a638611226af73240ee46ec74495ad576b2b46ecd015ea55cca1a8d09883d"}
Oct 29 23:14:02.221713 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:14:02.221681    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e66cd3_c0b8_4ce0_bd7f_dfddbdce9966.slice/crio-d3b6692448d189a717b5216e49c3775997192ff7f1de8cf305afe0753a216384 WatchSource:0}: Error finding container d3b6692448d189a717b5216e49c3775997192ff7f1de8cf305afe0753a216384: Status 404 returned error can't find the container with id d3b6692448d189a717b5216e49c3775997192ff7f1de8cf305afe0753a216384
Oct 29 23:14:02.223057 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:02.223032    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"]
Oct 29 23:14:02.896042 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:02.896007    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:03.173226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:03.173144    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg" event={"ID":"3b5c0e17-151f-4471-b12c-d7b7228bf80a","Type":"ContainerStarted","Data":"fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf"}
Oct 29 23:14:03.173643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:03.173233    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"
Oct 29 23:14:03.174811 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:03.174785    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw" event={"ID":"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966","Type":"ContainerStarted","Data":"4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755"}
Oct 29 23:14:03.175178 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:03.174831    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw" event={"ID":"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966","Type":"ContainerStarted","Data":"d3b6692448d189a717b5216e49c3775997192ff7f1de8cf305afe0753a216384"}
Oct 29 23:14:03.175178 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:03.174860    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"
Oct 29 23:14:03.216628 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:03.216588    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg" podStartSLOduration=2.2165775 podStartE2EDuration="2.2165775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:03.215171786 +0000 UTC m=+3585.239947936" watchObservedRunningTime="2026-10-29 23:14:03.2165775 +0000 UTC m=+3585.241353630"
Oct 29 23:14:03.256251 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:03.256212    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw" podStartSLOduration=2.25620065 podStartE2EDuration="2.25620065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:03.255144881 +0000 UTC m=+3585.279921011" watchObservedRunningTime="2026-10-29 23:14:03.25620065 +0000 UTC m=+3585.280976786"
Oct 29 23:14:04.037479 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:04.037444    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"]
Oct 29 23:14:04.050053 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:04.050022    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rhsbl"]
Oct 29 23:14:04.109923 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:04.109893    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901d5342-6569-4d10-8f0e-3bf34fa095a8" path="/var/lib/kubelet/pods/901d5342-6569-4d10-8f0e-3bf34fa095a8/volumes"
Oct 29 23:14:04.849608 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:04.849577    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:06.718271 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:06.718229    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:08.412482 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:08.412439    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:09.979828 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:09.979780    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:11.828997 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:11.828968    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:13.350369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:13.350304    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:14.181477 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:14.181441    4860 patch_prober.go:29] interesting pod/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:14:14.181658 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:14.181509    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:14.183219 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:14.183194    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"
Oct 29 23:14:14.183369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:14.183264    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"
Oct 29 23:14:14.526233 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:14.526202    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:15.199578 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:15.199547    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:15.253400 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:15.253370    4860 generic.go:356] "Generic (PLEG): container finished" podID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerID="ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33" exitCode=0
Oct 29 23:14:15.253557 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:15.253417    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" event={"ID":"c668e357-23fd-44b9-8aa3-e3aae2f061c8","Type":"ContainerDied","Data":"ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33"}
Oct 29 23:14:15.253812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:15.253793    4860 scope.go:122] "RemoveContainer" containerID="ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33"
Oct 29 23:14:15.255076 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:15.255043    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:14:15.255176 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:15.255116    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:14:15.255176 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:15.255154    4860 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:14:15.341137 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:15.341081    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:16.261349 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:16.261301    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" event={"ID":"c668e357-23fd-44b9-8aa3-e3aae2f061c8","Type":"ContainerStarted","Data":"1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72"}
Oct 29 23:14:16.812223 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:16.812195    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:17.010994 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.010959    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:17.237538 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.237510    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:14:17.237728 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.237702    4860 patch_prober.go:29] interesting pod/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.135:15021/healthz/ready\": dial tcp 10.132.1.135:15021: connect: connection refused" start-of-body=
Oct 29 23:14:17.237793 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.237743    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.135:15021/healthz/ready\": dial tcp 10.132.1.135:15021: connect: connection refused"
Oct 29 23:14:17.343117 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.343089    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:17.864688 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.864647    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"]
Oct 29 23:14:17.867306 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.867285    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:17.898100 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.898074    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-063169\"/\"kube-root-ca.crt\""
Oct 29 23:14:17.898911 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.898886    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:17.900860 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.900831    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"]
Oct 29 23:14:17.900981 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.900958    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-063169\"/\"openshift-service-ca.crt\""
Oct 29 23:14:17.901055 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.901017    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-063169\"/\"simulator-single-main-063169-kserve-self-signed-certs\""
Oct 29 23:14:17.901138 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:17.901120    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-063169\"/\"default-dockercfg-8zkx6\""
Oct 29 23:14:18.013788 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.013751    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-dshm\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.013980 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.013825    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tmp-dir\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.013980 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.013850    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tls-certs\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.013980 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.013931    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-home\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.013980 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.013961    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd7x9\" (UniqueName: \"kubernetes.io/projected/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-kube-api-access-rd7x9\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.014182 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.014046    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-model-cache\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.056998 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:14:18.056959    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:14:18.114563 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.114531    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd7x9\" (UniqueName: \"kubernetes.io/projected/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-kube-api-access-rd7x9\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.114768 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.114567    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-model-cache\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.114768 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.114636    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-dshm\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.114768 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.114654    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tmp-dir\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.114768 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.114674    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tls-certs\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.114768 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.114742    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-home\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.115098 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.115015    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-model-cache\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.115098 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.115071    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tmp-dir\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.115201 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.115128    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-home\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.116849 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.116828    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-dshm\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.117155 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.117135    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tls-certs\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.134975 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.134955    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd7x9\" (UniqueName: \"kubernetes.io/projected/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-kube-api-access-rd7x9\") pod \"simulator-single-main-063169-kserve-d5c668d5c-52smp\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") " pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.189697 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.189456    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:18.237241 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.237156    4860 patch_prober.go:29] interesting pod/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.135:15021/healthz/ready\": dial tcp 10.132.1.135:15021: connect: connection refused" start-of-body=
Oct 29 23:14:18.237241 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.237219    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.135:15021/healthz/ready\": dial tcp 10.132.1.135:15021: connect: connection refused"
Oct 29 23:14:18.503742 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:14:18.503700    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c5a6874_1c79_4d8d_ad57_867aa3b1c551.slice/crio-3fda4ad32a7944f917563d08e5dfea350dc0cf226099aad267eecc3a8011fb38 WatchSource:0}: Error finding container 3fda4ad32a7944f917563d08e5dfea350dc0cf226099aad267eecc3a8011fb38: Status 404 returned error can't find the container with id 3fda4ad32a7944f917563d08e5dfea350dc0cf226099aad267eecc3a8011fb38
Oct 29 23:14:18.505342 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.505183    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"]
Oct 29 23:14:18.903302 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.903215    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 23:14:18.922411 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:18.922388    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 23:14:19.238087 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:19.238054    4860 patch_prober.go:29] interesting pod/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.135:15021/healthz/ready\": dial tcp 10.132.1.135:15021: connect: connection refused" start-of-body=
Oct 29 23:14:19.238262 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:19.238108    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.135:15021/healthz/ready\": dial tcp 10.132.1.135:15021: connect: connection refused"
Oct 29 23:14:19.309104 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:19.309071    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp" event={"ID":"1c5a6874-1c79-4d8d-ad57-867aa3b1c551","Type":"ContainerStarted","Data":"d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21"}
Oct 29 23:14:19.309287 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:19.309113    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp" event={"ID":"1c5a6874-1c79-4d8d-ad57-867aa3b1c551","Type":"ContainerStarted","Data":"3fda4ad32a7944f917563d08e5dfea350dc0cf226099aad267eecc3a8011fb38"}
Oct 29 23:14:19.365812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:19.365773    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp" podStartSLOduration=2.36576093 podStartE2EDuration="2.36576093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:19.365007929 +0000 UTC m=+3601.389784060" watchObservedRunningTime="2026-10-29 23:14:19.36576093 +0000 UTC m=+3601.390537061"
Oct 29 23:14:19.817791 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:19.817745    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:20.192412 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:20.191413    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:20.237857 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:20.237825    4860 patch_prober.go:29] interesting pod/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.135:15021/healthz/ready\": dial tcp 10.132.1.135:15021: connect: connection refused" start-of-body=
Oct 29 23:14:20.238047 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:20.237878    4860 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.135:15021/healthz/ready\": dial tcp 10.132.1.135:15021: connect: connection refused"
Oct 29 23:14:20.422711 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:20.422672    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:21.029955 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:21.029920    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"]
Oct 29 23:14:21.041945 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:21.041914    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2cctc"]
Oct 29 23:14:21.601387 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:21.601360    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:14:21.601717 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:21.601695    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:14:21.602624 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:21.602604    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:14:21.986117 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:21.986082    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:22.110535 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:22.110506    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9" path="/var/lib/kubelet/pods/e6e434bb-2ee7-498a-8cf8-b90f9f2e7ff9/volumes"
Oct 29 23:14:23.701444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:23.701407    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:25.233550 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:25.233513    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:25.565793 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:25.565715    4860 scope.go:122] "RemoveContainer" containerID="029921de402b631b2c3bbd0dac042936ff508bab572b0ff8f5d6b92433dddaa0"
Oct 29 23:14:26.939820 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:26.939770    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:28.190864 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:28.190830    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:28.203643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:28.203620    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:28.380064 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:28.380035    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:14:28.511893 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:28.511811    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:30.099749 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:30.099718    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:31.788539 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:31.788505    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:32.644224 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:32.644193    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:32.762164 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:32.762131    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:33.420637 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:33.420594    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:33.963674 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:33.963640    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"]
Oct 29 23:14:33.968000 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:33.967980    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:33.997314 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:33.997282    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:14:33.997484 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:33.997392    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:14:33.997484 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:33.997412    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:14:33.997484 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:33.997466    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:14:34.000115 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:34.000094    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"]
Oct 29 23:14:34.061876 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:34.061849    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa147a01-dnv7m\" (UID: \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:34.061997 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:34.061946    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85zxp\" (UniqueName: \"kubernetes.io/projected/1693e04a-5d59-4749-8b3d-8d26373d9e1e-kube-api-access-85zxp\") pod \"maas-api-revoke-sub-aa147a01-dnv7m\" (UID: \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:34.062070 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:34.062042    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa147a01-dnv7m\" (UID: \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:34.162892 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:34.162860    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85zxp\" (UniqueName: \"kubernetes.io/projected/1693e04a-5d59-4749-8b3d-8d26373d9e1e-kube-api-access-85zxp\") pod \"maas-api-revoke-sub-aa147a01-dnv7m\" (UID: \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:34.163049 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:34.162945    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa147a01-dnv7m\" (UID: \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:34.163049 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:34.163027    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa147a01-dnv7m\" (UID: \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:34.163725 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:34.163700    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa147a01-dnv7m\" (UID: \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:34.165225 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:34.165202    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa147a01-dnv7m\" (UID: \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:34.187307 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:34.187284    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85zxp\" (UniqueName: \"kubernetes.io/projected/1693e04a-5d59-4749-8b3d-8d26373d9e1e-kube-api-access-85zxp\") pod \"maas-api-revoke-sub-aa147a01-dnv7m\" (UID: \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:34.281728 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:34.281638    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:34.590900 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:34.590859    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"]
Oct 29 23:14:34.593419 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:14:34.593382    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1693e04a_5d59_4749_8b3d_8d26373d9e1e.slice/crio-14b219eb8403503ae0e594bb30dc491a508093248bc69bbf87a7d43fd73c5da2 WatchSource:0}: Error finding container 14b219eb8403503ae0e594bb30dc491a508093248bc69bbf87a7d43fd73c5da2: Status 404 returned error can't find the container with id 14b219eb8403503ae0e594bb30dc491a508093248bc69bbf87a7d43fd73c5da2
Oct 29 23:14:35.284666 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:35.284639    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:35.413224 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:35.413197    4860 generic.go:356] "Generic (PLEG): container finished" podID="1693e04a-5d59-4749-8b3d-8d26373d9e1e" containerID="2ca9bda0aa0ebd85708bb1d3b93483110e10278e04e9b75485bc9b2af95f1ad8" exitCode=0
Oct 29 23:14:35.413390 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:35.413282    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m" event={"ID":"1693e04a-5d59-4749-8b3d-8d26373d9e1e","Type":"ContainerDied","Data":"2ca9bda0aa0ebd85708bb1d3b93483110e10278e04e9b75485bc9b2af95f1ad8"}
Oct 29 23:14:35.413390 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:35.413355    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m" event={"ID":"1693e04a-5d59-4749-8b3d-8d26373d9e1e","Type":"ContainerStarted","Data":"14b219eb8403503ae0e594bb30dc491a508093248bc69bbf87a7d43fd73c5da2"}
Oct 29 23:14:36.552122 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:36.552098    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:36.585486 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:36.585459    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1693e04a-5d59-4749-8b3d-8d26373d9e1e-kube-api-access-85zxp\" (UniqueName: \"kubernetes.io/projected/1693e04a-5d59-4749-8b3d-8d26373d9e1e-kube-api-access-85zxp\") pod \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\" (UID: \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\") "
Oct 29 23:14:36.585639 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:36.585555    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-openshift-ca\") pod \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\" (UID: \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\") "
Oct 29 23:14:36.585639 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:36.585619    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-xks-ca\") pod \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\" (UID: \"1693e04a-5d59-4749-8b3d-8d26373d9e1e\") "
Oct 29 23:14:36.585961 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:36.585934    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-openshift-ca" pod "1693e04a-5d59-4749-8b3d-8d26373d9e1e" (UID: "1693e04a-5d59-4749-8b3d-8d26373d9e1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:36.587570 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:36.587542    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1693e04a-5d59-4749-8b3d-8d26373d9e1e-kube-api-access-85zxp" pod "1693e04a-5d59-4749-8b3d-8d26373d9e1e" (UID: "1693e04a-5d59-4749-8b3d-8d26373d9e1e"). InnerVolumeSpecName "kube-api-access-85zxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:14:36.587666 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:36.587581    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-xks-ca" pod "1693e04a-5d59-4749-8b3d-8d26373d9e1e" (UID: "1693e04a-5d59-4749-8b3d-8d26373d9e1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:36.686693 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:36.686666    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:14:36.686693 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:36.686691    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1693e04a-5d59-4749-8b3d-8d26373d9e1e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:14:36.686897 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:36.686701    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85zxp\" (UniqueName: \"kubernetes.io/projected/1693e04a-5d59-4749-8b3d-8d26373d9e1e-kube-api-access-85zxp\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:14:36.920527 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:36.920491    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:37.425315 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:37.425288    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"
Oct 29 23:14:37.425516 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:37.425355    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m" event={"ID":"1693e04a-5d59-4749-8b3d-8d26373d9e1e","Type":"ContainerDied","Data":"14b219eb8403503ae0e594bb30dc491a508093248bc69bbf87a7d43fd73c5da2"}
Oct 29 23:14:37.425516 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:37.425412    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14b219eb8403503ae0e594bb30dc491a508093248bc69bbf87a7d43fd73c5da2"
Oct 29 23:14:38.605892 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:38.605757    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:40.187338 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:40.187295    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:41.874140 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:41.874108    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:43.467629 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:43.467595    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:44.760143 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.760113    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:44.910762 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.910731    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"]
Oct 29 23:14:44.911183 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.911169    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1693e04a-5d59-4749-8b3d-8d26373d9e1e" containerName="revoke-keys"
Oct 29 23:14:44.911238 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.911184    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1693e04a-5d59-4749-8b3d-8d26373d9e1e" containerName="revoke-keys"
Oct 29 23:14:44.911285 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.911264    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1693e04a-5d59-4749-8b3d-8d26373d9e1e" containerName="revoke-keys"
Oct 29 23:14:44.913545 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.913522    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:44.943507 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.943483    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:14:44.943653 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.943531    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:14:44.943653 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.943549    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:14:44.943653 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.943586    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:14:44.946398 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.946375    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"]
Oct 29 23:14:44.955511 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.955485    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663b3214-09ae-417d-b902-034df968c18e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w\" (UID: \"663b3214-09ae-417d-b902-034df968c18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:44.955634 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.955538    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663b3214-09ae-417d-b902-034df968c18e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w\" (UID: \"663b3214-09ae-417d-b902-034df968c18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:44.955634 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:44.955614    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx56s\" (UniqueName: \"kubernetes.io/projected/663b3214-09ae-417d-b902-034df968c18e-kube-api-access-bx56s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w\" (UID: \"663b3214-09ae-417d-b902-034df968c18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:45.042343 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:45.042261    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:45.056269 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:45.056246    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663b3214-09ae-417d-b902-034df968c18e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w\" (UID: \"663b3214-09ae-417d-b902-034df968c18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:45.056417 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:45.056281    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663b3214-09ae-417d-b902-034df968c18e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w\" (UID: \"663b3214-09ae-417d-b902-034df968c18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:45.056417 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:45.056301    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx56s\" (UniqueName: \"kubernetes.io/projected/663b3214-09ae-417d-b902-034df968c18e-kube-api-access-bx56s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w\" (UID: \"663b3214-09ae-417d-b902-034df968c18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:45.057041 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:45.057020    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663b3214-09ae-417d-b902-034df968c18e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w\" (UID: \"663b3214-09ae-417d-b902-034df968c18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:45.058923 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:45.058878    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663b3214-09ae-417d-b902-034df968c18e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w\" (UID: \"663b3214-09ae-417d-b902-034df968c18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:45.076396 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:45.076371    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx56s\" (UniqueName: \"kubernetes.io/projected/663b3214-09ae-417d-b902-034df968c18e-kube-api-access-bx56s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w\" (UID: \"663b3214-09ae-417d-b902-034df968c18e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:45.225696 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:45.225666    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:45.521556 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:14:45.521522    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663b3214_09ae_417d_b902_034df968c18e.slice/crio-d3182e431067ff2e79f2d3eec8ca9bbd94c9796771629cace24950cb4f92670c WatchSource:0}: Error finding container d3182e431067ff2e79f2d3eec8ca9bbd94c9796771629cace24950cb4f92670c: Status 404 returned error can't find the container with id d3182e431067ff2e79f2d3eec8ca9bbd94c9796771629cace24950cb4f92670c
Oct 29 23:14:45.523084 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:45.523018    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"]
Oct 29 23:14:46.479036 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:46.479000    4860 generic.go:356] "Generic (PLEG): container finished" podID="663b3214-09ae-417d-b902-034df968c18e" containerID="dd732238812243ffe351167b34b2ca09fdfd16ef5af908aab383996d7440184e" exitCode=0
Oct 29 23:14:46.479481 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:46.479084    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w" event={"ID":"663b3214-09ae-417d-b902-034df968c18e","Type":"ContainerDied","Data":"dd732238812243ffe351167b34b2ca09fdfd16ef5af908aab383996d7440184e"}
Oct 29 23:14:46.479481 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:46.479134    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w" event={"ID":"663b3214-09ae-417d-b902-034df968c18e","Type":"ContainerStarted","Data":"d3182e431067ff2e79f2d3eec8ca9bbd94c9796771629cace24950cb4f92670c"}
Oct 29 23:14:46.840397 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:46.840337    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:47.620020 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:47.619995    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:47.679207 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:47.679179    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663b3214-09ae-417d-b902-034df968c18e-kube-api-access-bx56s\" (UniqueName: \"kubernetes.io/projected/663b3214-09ae-417d-b902-034df968c18e-kube-api-access-bx56s\") pod \"663b3214-09ae-417d-b902-034df968c18e\" (UID: \"663b3214-09ae-417d-b902-034df968c18e\") "
Oct 29 23:14:47.679368 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:47.679303    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663b3214-09ae-417d-b902-034df968c18e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663b3214-09ae-417d-b902-034df968c18e-maas-api-openshift-ca\") pod \"663b3214-09ae-417d-b902-034df968c18e\" (UID: \"663b3214-09ae-417d-b902-034df968c18e\") "
Oct 29 23:14:47.679368 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:47.679360    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/663b3214-09ae-417d-b902-034df968c18e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663b3214-09ae-417d-b902-034df968c18e-maas-api-xks-ca\") pod \"663b3214-09ae-417d-b902-034df968c18e\" (UID: \"663b3214-09ae-417d-b902-034df968c18e\") "
Oct 29 23:14:47.679658 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:47.679634    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663b3214-09ae-417d-b902-034df968c18e-maas-api-openshift-ca" pod "663b3214-09ae-417d-b902-034df968c18e" (UID: "663b3214-09ae-417d-b902-034df968c18e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:47.679785 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:47.679768    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663b3214-09ae-417d-b902-034df968c18e-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:14:47.681297 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:47.681270    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663b3214-09ae-417d-b902-034df968c18e-maas-api-xks-ca" pod "663b3214-09ae-417d-b902-034df968c18e" (UID: "663b3214-09ae-417d-b902-034df968c18e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:47.681463 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:47.681443    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663b3214-09ae-417d-b902-034df968c18e-kube-api-access-bx56s" pod "663b3214-09ae-417d-b902-034df968c18e" (UID: "663b3214-09ae-417d-b902-034df968c18e"). InnerVolumeSpecName "kube-api-access-bx56s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:14:47.781158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:47.781081    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx56s\" (UniqueName: \"kubernetes.io/projected/663b3214-09ae-417d-b902-034df968c18e-kube-api-access-bx56s\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:14:47.781158 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:47.781110    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663b3214-09ae-417d-b902-034df968c18e-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:14:48.057223 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:14:48.057137    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:14:48.466444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:48.466415    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:48.493006 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:48.492981    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"
Oct 29 23:14:48.494375 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:48.493015    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w" event={"ID":"663b3214-09ae-417d-b902-034df968c18e","Type":"ContainerDied","Data":"d3182e431067ff2e79f2d3eec8ca9bbd94c9796771629cace24950cb4f92670c"}
Oct 29 23:14:48.494609 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:48.494402    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3182e431067ff2e79f2d3eec8ca9bbd94c9796771629cace24950cb4f92670c"
Oct 29 23:14:49.700970 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:49.700940    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"]
Oct 29 23:14:49.790619 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:49.790585    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:50.099068 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:50.098984    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:51.692822 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:51.692786    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:53.188580 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:53.188549    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:55.009291 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:55.009256    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:55.367479 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:55.367409    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"]
Oct 29 23:14:55.401176 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:55.401149    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"]
Oct 29 23:14:55.656606 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:55.656509    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:55.783081 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:55.783048    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:56.615951 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:56.615910    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:57.953142 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:57.953110    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:58.573898 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:58.573866    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:58.684515 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:58.684482    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"]
Oct 29 23:14:58.687030 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:58.687004    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-063169-ccx8w"]
Oct 29 23:14:58.715315 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:58.715285    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:59.242387 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:14:59.242357    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:00.109782 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:00.109748    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663b3214-09ae-417d-b902-034df968c18e" path="/var/lib/kubelet/pods/663b3214-09ae-417d-b902-034df968c18e/volumes"
Oct 29 23:15:00.367865 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:00.367774    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw" podUID="c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966" containerName="payload-processing" containerID="cri-o://4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755" gracePeriod=25
Oct 29 23:15:00.402020 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:00.401984    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg" podUID="3b5c0e17-151f-4471-b12c-d7b7228bf80a" containerName="payload-pre-processing" containerID="cri-o://fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf" gracePeriod=25
Oct 29 23:15:04.182398 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:04.182361    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw" podUID="c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:15:04.182761 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:04.182402    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg" podUID="3b5c0e17-151f-4471-b12c-d7b7228bf80a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:15:04.721682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:04.721636    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp" podUID="1c5a6874-1c79-4d8d-ad57-867aa3b1c551" containerName="main" containerID="cri-o://d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21" gracePeriod=45
Oct 29 23:15:04.965208 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:04.965187    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:15:05.135234 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.135158    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-dshm\") pod \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") "
Oct 29 23:15:05.135234 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.135203    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tmp-dir\") pod \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") "
Oct 29 23:15:05.135498 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.135254    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-kube-api-access-rd7x9\" (UniqueName: \"kubernetes.io/projected/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-kube-api-access-rd7x9\") pod \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") "
Oct 29 23:15:05.135498 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.135276    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tls-certs\") pod \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") "
Oct 29 23:15:05.135498 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.135377    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-home\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-home\") pod \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") "
Oct 29 23:15:05.135498 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.135399    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-model-cache\") pod \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\" (UID: \"1c5a6874-1c79-4d8d-ad57-867aa3b1c551\") "
Oct 29 23:15:05.135694 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.135499    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tmp-dir" pod "1c5a6874-1c79-4d8d-ad57-867aa3b1c551" (UID: "1c5a6874-1c79-4d8d-ad57-867aa3b1c551"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:05.135748 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.135678    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-model-cache" pod "1c5a6874-1c79-4d8d-ad57-867aa3b1c551" (UID: "1c5a6874-1c79-4d8d-ad57-867aa3b1c551"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:05.135748 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.135691    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-home" pod "1c5a6874-1c79-4d8d-ad57-867aa3b1c551" (UID: "1c5a6874-1c79-4d8d-ad57-867aa3b1c551"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:05.135825 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.135764    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:05.137585 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.137561    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tls-certs" pod "1c5a6874-1c79-4d8d-ad57-867aa3b1c551" (UID: "1c5a6874-1c79-4d8d-ad57-867aa3b1c551"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:15:05.137839 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.137823    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-dshm" pod "1c5a6874-1c79-4d8d-ad57-867aa3b1c551" (UID: "1c5a6874-1c79-4d8d-ad57-867aa3b1c551"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:05.137896 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.137874    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-kube-api-access-rd7x9" pod "1c5a6874-1c79-4d8d-ad57-867aa3b1c551" (UID: "1c5a6874-1c79-4d8d-ad57-867aa3b1c551"). InnerVolumeSpecName "kube-api-access-rd7x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:05.236817 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.236792    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd7x9\" (UniqueName: \"kubernetes.io/projected/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-kube-api-access-rd7x9\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:05.236817 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.236814    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:05.237190 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.236826    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:05.237190 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.236834    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:05.237190 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.236842    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c5a6874-1c79-4d8d-ad57-867aa3b1c551-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:05.419877 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.419820    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:05.554166 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.554125    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:05.592828 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.592789    4860 generic.go:356] "Generic (PLEG): container finished" podID="1c5a6874-1c79-4d8d-ad57-867aa3b1c551" containerID="d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21" exitCode=0
Oct 29 23:15:05.593016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.592838    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"
Oct 29 23:15:05.593016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.592864    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp" event={"ID":"1c5a6874-1c79-4d8d-ad57-867aa3b1c551","Type":"ContainerDied","Data":"d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21"}
Oct 29 23:15:05.593016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.592906    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp" event={"ID":"1c5a6874-1c79-4d8d-ad57-867aa3b1c551","Type":"ContainerDied","Data":"3fda4ad32a7944f917563d08e5dfea350dc0cf226099aad267eecc3a8011fb38"}
Oct 29 23:15:05.593016 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.592923    4860 scope.go:122] "RemoveContainer" containerID="d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21"
Oct 29 23:15:05.603218 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.603199    4860 scope.go:122] "RemoveContainer" containerID="d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21"
Oct 29 23:15:05.603555 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:15:05.603536    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21\": container with ID starting with d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21 not found: ID does not exist" containerID="d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21"
Oct 29 23:15:05.603620 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.603564    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21"} err="failed to get container status \"d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21\": rpc error: code = NotFound desc = could not find container \"d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21\": container with ID starting with d3d9c9d9452b4b0853b4df409e5bc7529188e6ba9a9c5b6c399dac2224b07e21 not found: ID does not exist"
Oct 29 23:15:05.647392 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.647369    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"]
Oct 29 23:15:05.660228 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.660207    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-063169/simulator-single-main-063169-kserve-d5c668d5c-52smp"]
Oct 29 23:15:05.714593 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.714564    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"]
Oct 29 23:15:05.714819 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.714789    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy" containerID="cri-o://1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72" gracePeriod=30
Oct 29 23:15:05.767251 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:05.767225    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:06.110537 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:06.110460    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5a6874-1c79-4d8d-ad57-867aa3b1c551" path="/var/lib/kubelet/pods/1c5a6874-1c79-4d8d-ad57-867aa3b1c551/volumes"
Oct 29 23:15:06.604526 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:06.604488    4860 patch_prober.go:29] interesting pod/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:15:06.604930 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:06.604566    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:15:10.970991 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.970962    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:15:10.977060 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.976990    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"]
Oct 29 23:15:10.977519 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.977500    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy"
Oct 29 23:15:10.977646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.977521    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy"
Oct 29 23:15:10.977646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.977539    4860 cpu_manager.go:414] "containerMap: removing container" podUID="1c5a6874-1c79-4d8d-ad57-867aa3b1c551" containerName="main"
Oct 29 23:15:10.977646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.977546    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5a6874-1c79-4d8d-ad57-867aa3b1c551" containerName="main"
Oct 29 23:15:10.977646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.977575    4860 cpu_manager.go:414] "containerMap: removing container" podUID="663b3214-09ae-417d-b902-034df968c18e" containerName="revoke-keys"
Oct 29 23:15:10.977646 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.977583    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663b3214-09ae-417d-b902-034df968c18e" containerName="revoke-keys"
Oct 29 23:15:10.977940 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.977701    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy"
Oct 29 23:15:10.977940 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.977714    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5a6874-1c79-4d8d-ad57-867aa3b1c551" containerName="main"
Oct 29 23:15:10.977940 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.977726    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy"
Oct 29 23:15:10.977940 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.977738    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="663b3214-09ae-417d-b902-034df968c18e" containerName="revoke-keys"
Oct 29 23:15:10.979964 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:10.979945    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.001421 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.001397    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"]
Oct 29 23:15:11.014823 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.014800    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-1fe7678b-kserve-self-signed-certs\""
Oct 29 23:15:11.072195 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.072162    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:11.092313 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092286    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-envoy\") pod \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") "
Oct 29 23:15:11.092434 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092336    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-credential-socket\") pod \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") "
Oct 29 23:15:11.092434 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092380    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istiod-ca-cert\") pod \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") "
Oct 29 23:15:11.092434 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092404    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-podinfo\") pod \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") "
Oct 29 23:15:11.092612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092432    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-certs\") pod \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") "
Oct 29 23:15:11.092612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092459    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-token\" (UniqueName: \"kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-token\") pod \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") "
Oct 29 23:15:11.092612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092531    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-socket\") pod \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") "
Oct 29 23:15:11.092612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092598    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-data\") pod \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") "
Oct 29 23:15:11.092857 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092644    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-kube-api-access-krbrj\" (UniqueName: \"kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-kube-api-access-krbrj\") pod \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\" (UID: \"c668e357-23fd-44b9-8aa3-e3aae2f061c8\") "
Oct 29 23:15:11.092857 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092721    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-credential-socket" pod "c668e357-23fd-44b9-8aa3-e3aae2f061c8" (UID: "c668e357-23fd-44b9-8aa3-e3aae2f061c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:11.092857 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092752    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-certs" pod "c668e357-23fd-44b9-8aa3-e3aae2f061c8" (UID: "c668e357-23fd-44b9-8aa3-e3aae2f061c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:11.092857 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092784    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-model-cache\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.092857 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092819    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istiod-ca-cert" pod "c668e357-23fd-44b9-8aa3-e3aae2f061c8" (UID: "c668e357-23fd-44b9-8aa3-e3aae2f061c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:11.092857 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092840    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-dshm\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.093226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092969    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsd9k\" (UniqueName: \"kubernetes.io/projected/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-kube-api-access-zsd9k\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.093226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.092976    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-data" pod "c668e357-23fd-44b9-8aa3-e3aae2f061c8" (UID: "c668e357-23fd-44b9-8aa3-e3aae2f061c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:11.093226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.093120    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-home\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.093226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.093224    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tls-certs\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.093509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.093255    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tmp-dir\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.093509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.093293    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-socket" pod "c668e357-23fd-44b9-8aa3-e3aae2f061c8" (UID: "c668e357-23fd-44b9-8aa3-e3aae2f061c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:11.093509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.093425    4860 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-data\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:11.093509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.093442    4860 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-credential-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:11.093509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.093452    4860 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istiod-ca-cert\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:11.093509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.093462    4860 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:11.093509 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.093470    4860 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-workload-socket\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:11.094760 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.094729    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-podinfo" pod "c668e357-23fd-44b9-8aa3-e3aae2f061c8" (UID: "c668e357-23fd-44b9-8aa3-e3aae2f061c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:15:11.094964 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.094907    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-token" pod "c668e357-23fd-44b9-8aa3-e3aae2f061c8" (UID: "c668e357-23fd-44b9-8aa3-e3aae2f061c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:11.095079 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.094964    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-envoy" pod "c668e357-23fd-44b9-8aa3-e3aae2f061c8" (UID: "c668e357-23fd-44b9-8aa3-e3aae2f061c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:15:11.095339 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.095306    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-kube-api-access-krbrj" pod "c668e357-23fd-44b9-8aa3-e3aae2f061c8" (UID: "c668e357-23fd-44b9-8aa3-e3aae2f061c8"). InnerVolumeSpecName "kube-api-access-krbrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:11.194640 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.194603    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsd9k\" (UniqueName: \"kubernetes.io/projected/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-kube-api-access-zsd9k\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.194811 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.194737    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-home\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.194811 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.194791    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tls-certs\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.194997 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.194839    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tmp-dir\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.194997 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.194900    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-model-cache\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.194997 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.194947    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-dshm\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.195138 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.195079    4860 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-podinfo\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:11.195245 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.195194    4860 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-token\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:11.195245 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.195219    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krbrj\" (UniqueName: \"kubernetes.io/projected/c668e357-23fd-44b9-8aa3-e3aae2f061c8-kube-api-access-krbrj\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:11.195245 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.195235    4860 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c668e357-23fd-44b9-8aa3-e3aae2f061c8-istio-envoy\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:11.195245 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.195196    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-home\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.195430 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.195254    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tmp-dir\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.195430 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.195394    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-model-cache\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.197236 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.197218    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-dshm\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.197558 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.197540    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tls-certs\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.214670 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.214640    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsd9k\" (UniqueName: \"kubernetes.io/projected/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-kube-api-access-zsd9k\") pod \"e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") " pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.294450 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.294375    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:11.585109 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:15:11.585079    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18e76f05_d6fc_4d40_b0a9_58f976b8ea70.slice/crio-4b45eda40c0325001dfb97ff0b246a42376c641e472f86aefee837149395bce9 WatchSource:0}: Error finding container 4b45eda40c0325001dfb97ff0b246a42376c641e472f86aefee837149395bce9: Status 404 returned error can't find the container with id 4b45eda40c0325001dfb97ff0b246a42376c641e472f86aefee837149395bce9
Oct 29 23:15:11.587431 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.587407    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"]
Oct 29 23:15:11.629817 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.629794    4860 generic.go:356] "Generic (PLEG): container finished" podID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerID="1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72" exitCode=0
Oct 29 23:15:11.629928 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.629850    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"
Oct 29 23:15:11.629928 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.629880    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" event={"ID":"c668e357-23fd-44b9-8aa3-e3aae2f061c8","Type":"ContainerDied","Data":"1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72"}
Oct 29 23:15:11.630028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.629938    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk" event={"ID":"c668e357-23fd-44b9-8aa3-e3aae2f061c8","Type":"ContainerDied","Data":"7cf04a0ef5540f0c0c93ce823ec751787229bbc133532574130b27da99a29cfe"}
Oct 29 23:15:11.630028 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.629969    4860 scope.go:122] "RemoveContainer" containerID="1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72"
Oct 29 23:15:11.631514 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.631490    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq" event={"ID":"18e76f05-d6fc-4d40-b0a9-58f976b8ea70","Type":"ContainerStarted","Data":"4b45eda40c0325001dfb97ff0b246a42376c641e472f86aefee837149395bce9"}
Oct 29 23:15:11.642627 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.642451    4860 scope.go:122] "RemoveContainer" containerID="ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33"
Oct 29 23:15:11.651299 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.651282    4860 scope.go:122] "RemoveContainer" containerID="1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72"
Oct 29 23:15:11.651571 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:15:11.651553    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72\": container with ID starting with 1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72 not found: ID does not exist" containerID="1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72"
Oct 29 23:15:11.651620 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.651579    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72"} err="failed to get container status \"1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72\": rpc error: code = NotFound desc = could not find container \"1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72\": container with ID starting with 1002f22af3e6c1b38c9624ca33fee4ecf606df1103340969164da87362a86f72 not found: ID does not exist"
Oct 29 23:15:11.651620 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.651596    4860 scope.go:122] "RemoveContainer" containerID="ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33"
Oct 29 23:15:11.651824 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:15:11.651808    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33\": container with ID starting with ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33 not found: ID does not exist" containerID="ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33"
Oct 29 23:15:11.651878 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.651829    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33"} err="failed to get container status \"ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33\": rpc error: code = NotFound desc = could not find container \"ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33\": container with ID starting with ad464d2aa1f8b106e6fd5651d209cc14532ad689322bed18541fe0acc59ace33 not found: ID does not exist"
Oct 29 23:15:11.685743 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.685714    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"]
Oct 29 23:15:11.698377 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:11.698353    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-063169-openshift-default-786ffd57b5prdhk"]
Oct 29 23:15:12.110026 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:12.109999    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" path="/var/lib/kubelet/pods/c668e357-23fd-44b9-8aa3-e3aae2f061c8/volumes"
Oct 29 23:15:12.638503 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:12.638468    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq" event={"ID":"18e76f05-d6fc-4d40-b0a9-58f976b8ea70","Type":"ContainerStarted","Data":"fb75fa032002da57c6dc914ee67893865fd8468dbda8bbcca64a4f9a380e1841"}
Oct 29 23:15:12.682408 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:12.682353    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq" podStartSLOduration=2.682337316 podStartE2EDuration="2.682337316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:12.680197762 +0000 UTC m=+3654.704973892" watchObservedRunningTime="2026-10-29 23:15:12.682337316 +0000 UTC m=+3654.707113448"
Oct 29 23:15:13.295115 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:13.295082    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:13.454267 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:13.454236    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:13.573737 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:13.573662    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:14.182729 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:14.182694    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw" podUID="c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:15:14.183066 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:14.182774    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg" podUID="3b5c0e17-151f-4471-b12c-d7b7228bf80a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:15:16.919070 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:16.919035    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:17.259762 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:17.259730    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:18.056251 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:15:18.056218    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:15:20.653345 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.653301    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"
Oct 29 23:15:20.679739 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.679711    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"
Oct 29 23:15:20.691560 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.691532    4860 generic.go:356] "Generic (PLEG): container finished" podID="3b5c0e17-151f-4471-b12c-d7b7228bf80a" containerID="fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf" exitCode=0
Oct 29 23:15:20.691686 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.691580    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"
Oct 29 23:15:20.691686 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.691622    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg" event={"ID":"3b5c0e17-151f-4471-b12c-d7b7228bf80a","Type":"ContainerDied","Data":"fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf"}
Oct 29 23:15:20.691686 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.691676    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg" event={"ID":"3b5c0e17-151f-4471-b12c-d7b7228bf80a","Type":"ContainerDied","Data":"299a638611226af73240ee46ec74495ad576b2b46ecd015ea55cca1a8d09883d"}
Oct 29 23:15:20.691851 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.691701    4860 scope.go:122] "RemoveContainer" containerID="fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf"
Oct 29 23:15:20.692977 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.692950    4860 generic.go:356] "Generic (PLEG): container finished" podID="c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966" containerID="4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755" exitCode=0
Oct 29 23:15:20.693089 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.692996    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"
Oct 29 23:15:20.693089 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.693077    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw" event={"ID":"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966","Type":"ContainerDied","Data":"4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755"}
Oct 29 23:15:20.693318 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.693292    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw" event={"ID":"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966","Type":"ContainerDied","Data":"d3b6692448d189a717b5216e49c3775997192ff7f1de8cf305afe0753a216384"}
Oct 29 23:15:20.702051 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.702028    4860 scope.go:122] "RemoveContainer" containerID="fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf"
Oct 29 23:15:20.702363 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:15:20.702315    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf\": container with ID starting with fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf not found: ID does not exist" containerID="fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf"
Oct 29 23:15:20.702461 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.702375    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf"} err="failed to get container status \"fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf\": rpc error: code = NotFound desc = could not find container \"fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf\": container with ID starting with fc95571c2ac1fef90a554536494cb3b651f521633249a296070f79f3fc3f79cf not found: ID does not exist"
Oct 29 23:15:20.702461 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.702399    4860 scope.go:122] "RemoveContainer" containerID="4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755"
Oct 29 23:15:20.712231 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.712211    4860 scope.go:122] "RemoveContainer" containerID="4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755"
Oct 29 23:15:20.712547 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:15:20.712515    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755\": container with ID starting with 4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755 not found: ID does not exist" containerID="4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755"
Oct 29 23:15:20.712612 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.712556    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755"} err="failed to get container status \"4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755\": rpc error: code = NotFound desc = could not find container \"4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755\": container with ID starting with 4b2ea23abe122c7c5f71570c2e8fcb8c732877cf704697b2d4d034cc3eba6755 not found: ID does not exist"
Oct 29 23:15:20.789641 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.789614    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-plugins-config-volume\") pod \"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966\" (UID: \"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966\") "
Oct 29 23:15:20.789768 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.789676    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-kube-api-access-gzkr7\" (UniqueName: \"kubernetes.io/projected/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-kube-api-access-gzkr7\") pod \"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966\" (UID: \"c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966\") "
Oct 29 23:15:20.789768 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.789706    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5c0e17-151f-4471-b12c-d7b7228bf80a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5c0e17-151f-4471-b12c-d7b7228bf80a-plugins-config-volume\") pod \"3b5c0e17-151f-4471-b12c-d7b7228bf80a\" (UID: \"3b5c0e17-151f-4471-b12c-d7b7228bf80a\") "
Oct 29 23:15:20.789768 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.789746    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5c0e17-151f-4471-b12c-d7b7228bf80a-kube-api-access-lbdr4\" (UniqueName: \"kubernetes.io/projected/3b5c0e17-151f-4471-b12c-d7b7228bf80a-kube-api-access-lbdr4\") pod \"3b5c0e17-151f-4471-b12c-d7b7228bf80a\" (UID: \"3b5c0e17-151f-4471-b12c-d7b7228bf80a\") "
Oct 29 23:15:20.790075 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.790048    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-plugins-config-volume" pod "c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966" (UID: "c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:20.790075 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.790061    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5c0e17-151f-4471-b12c-d7b7228bf80a-plugins-config-volume" pod "3b5c0e17-151f-4471-b12c-d7b7228bf80a" (UID: "3b5c0e17-151f-4471-b12c-d7b7228bf80a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:15:20.791804 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.791782    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-kube-api-access-gzkr7" pod "c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966" (UID: "c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966"). InnerVolumeSpecName "kube-api-access-gzkr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:20.791979 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.791962    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5c0e17-151f-4471-b12c-d7b7228bf80a-kube-api-access-lbdr4" pod "3b5c0e17-151f-4471-b12c-d7b7228bf80a" (UID: "3b5c0e17-151f-4471-b12c-d7b7228bf80a"). InnerVolumeSpecName "kube-api-access-lbdr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:15:20.890643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.890592    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5c0e17-151f-4471-b12c-d7b7228bf80a-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:20.890643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.890615    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbdr4\" (UniqueName: \"kubernetes.io/projected/3b5c0e17-151f-4471-b12c-d7b7228bf80a-kube-api-access-lbdr4\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:20.890643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.890623    4860 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-plugins-config-volume\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:20.890643 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:20.890633    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzkr7\" (UniqueName: \"kubernetes.io/projected/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966-kube-api-access-gzkr7\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:15:21.040433 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:21.040401    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"]
Oct 29 23:15:21.053443 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:21.053413    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-063169-78d58mpvtg"]
Oct 29 23:15:21.079917 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:21.079891    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"]
Oct 29 23:15:21.084838 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:21.084814    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-063169-fbc96c98b5zgjw"]
Oct 29 23:15:21.294882 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:21.294852    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:21.306894 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:21.306867    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:21.711686 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:21.711652    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:15:22.111140 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:22.111064    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5c0e17-151f-4471-b12c-d7b7228bf80a" path="/var/lib/kubelet/pods/3b5c0e17-151f-4471-b12c-d7b7228bf80a/volumes"
Oct 29 23:15:22.111512 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:22.111496    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966" path="/var/lib/kubelet/pods/c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966/volumes"
Oct 29 23:15:25.589897 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:25.589861    4860 scope.go:122] "RemoveContainer" containerID="6bbed58a04e1b6116511ee48647be41a978a6fe3d7cba26b3d8651d4c8b66849"
Oct 29 23:15:25.598949 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:25.598930    4860 scope.go:122] "RemoveContainer" containerID="15252cf7ef751ed5abb2cbb446bd4b6b1a8f3689211f8166c5f83f3c12dc461c"
Oct 29 23:15:25.606977 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:25.606959    4860 scope.go:122] "RemoveContainer" containerID="e0f943c1ac33cb9bbcbb7c79db508b4a10586f576cdeba9ab1eda240dcf42ac3"
Oct 29 23:15:26.713082 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:26.713050    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:26.970225 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:26.970144    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"]
Oct 29 23:15:26.970680 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:26.970660    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966" containerName="payload-processing"
Oct 29 23:15:26.970791 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:26.970682    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966" containerName="payload-processing"
Oct 29 23:15:26.970791 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:26.970705    4860 cpu_manager.go:414] "containerMap: removing container" podUID="3b5c0e17-151f-4471-b12c-d7b7228bf80a" containerName="payload-pre-processing"
Oct 29 23:15:26.970791 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:26.970714    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5c0e17-151f-4471-b12c-d7b7228bf80a" containerName="payload-pre-processing"
Oct 29 23:15:26.970791 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:26.970737    4860 cpu_manager.go:414] "containerMap: removing container" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy"
Oct 29 23:15:26.970791 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:26.970744    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c668e357-23fd-44b9-8aa3-e3aae2f061c8" containerName="istio-proxy"
Oct 29 23:15:26.971082 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:26.970887    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e66cd3-c0b8-4ce0-bd7f-dfddbdce9966" containerName="payload-processing"
Oct 29 23:15:26.971082 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:26.970901    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5c0e17-151f-4471-b12c-d7b7228bf80a" containerName="payload-pre-processing"
Oct 29 23:15:26.975302 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:26.975276    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:26.991130 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:26.991108    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"]
Oct 29 23:15:27.001993 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.001962    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-1fe7678b-kserve-self-signed-certs\""
Oct 29 23:15:27.033448 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.033423    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-model-cache\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.033571 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.033469    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-dshm\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.033571 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.033517    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tls-certs\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.033571 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.033544    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-home\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.033697 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.033577    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tmp-dir\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.033697 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.033671    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qlx\" (UniqueName: \"kubernetes.io/projected/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-kube-api-access-r2qlx\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.046843 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.046824    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:27.134133 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.134108    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tmp-dir\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.134258 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.134140    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qlx\" (UniqueName: \"kubernetes.io/projected/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-kube-api-access-r2qlx\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.134309 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.134288    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-model-cache\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.134596 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.134559    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-dshm\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.134719 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.134696    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tls-certs\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.134802 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.134756    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-home\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.134802 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.134775    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-model-cache\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.134923 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.134822    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tmp-dir\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.135013 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.134996    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-home\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.136729 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.136701    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-dshm\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.137145 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.137126    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tls-certs\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.155674 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.155650    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qlx\" (UniqueName: \"kubernetes.io/projected/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-kube-api-access-r2qlx\") pod \"e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") " pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.286842 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.286762    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:27.383601 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.383572    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:27.580223 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:15:27.580194    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod373a3fda_5bdf_49b6_b0e2_250d55c0d1f3.slice/crio-945a8bcb4a7f413e8ddeae54bd6508b1d6e51ab84d97fb90961415e8fc01a67d WatchSource:0}: Error finding container 945a8bcb4a7f413e8ddeae54bd6508b1d6e51ab84d97fb90961415e8fc01a67d: Status 404 returned error can't find the container with id 945a8bcb4a7f413e8ddeae54bd6508b1d6e51ab84d97fb90961415e8fc01a67d
Oct 29 23:15:27.580887 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.580859    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"]
Oct 29 23:15:27.736677 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.736643    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx" event={"ID":"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3","Type":"ContainerStarted","Data":"52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6"}
Oct 29 23:15:27.737052 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.736685    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx" event={"ID":"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3","Type":"ContainerStarted","Data":"945a8bcb4a7f413e8ddeae54bd6508b1d6e51ab84d97fb90961415e8fc01a67d"}
Oct 29 23:15:27.782020 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:27.781974    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx" podStartSLOduration=1.781961521 podStartE2EDuration="1.781961521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:27.780398298 +0000 UTC m=+3669.805174429" watchObservedRunningTime="2026-10-29 23:15:27.781961521 +0000 UTC m=+3669.806737703"
Oct 29 23:15:28.287080 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:28.287047    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:32.887257 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:32.887221    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:37.287480 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:37.287441    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:37.299342 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:37.299302    4860 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:37.817850 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:37.817824    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:15:44.585433 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:44.585399    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:48.059830 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:15:48.059747    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:15:56.781177 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:56.781141    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"]
Oct 29 23:15:56.863247 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:15:56.863215    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:01.057723 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:01.057682    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"]
Oct 29 23:16:01.065563 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:01.065521    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:11.801731 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:11.801672    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq" podUID="18e76f05-d6fc-4d40-b0a9-58f976b8ea70" containerName="main" containerID="cri-o://fb75fa032002da57c6dc914ee67893865fd8468dbda8bbcca64a4f9a380e1841" gracePeriod=45
Oct 29 23:16:12.039804 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.039779    4860 generic.go:356] "Generic (PLEG): container finished" podID="18e76f05-d6fc-4d40-b0a9-58f976b8ea70" containerID="fb75fa032002da57c6dc914ee67893865fd8468dbda8bbcca64a4f9a380e1841" exitCode=0
Oct 29 23:16:12.039906 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.039848    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq" event={"ID":"18e76f05-d6fc-4d40-b0a9-58f976b8ea70","Type":"ContainerDied","Data":"fb75fa032002da57c6dc914ee67893865fd8468dbda8bbcca64a4f9a380e1841"}
Oct 29 23:16:12.039906 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.039887    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq" event={"ID":"18e76f05-d6fc-4d40-b0a9-58f976b8ea70","Type":"ContainerDied","Data":"4b45eda40c0325001dfb97ff0b246a42376c641e472f86aefee837149395bce9"}
Oct 29 23:16:12.039906 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.039899    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b45eda40c0325001dfb97ff0b246a42376c641e472f86aefee837149395bce9"
Oct 29 23:16:12.048543 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.048520    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:16:12.075892 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.075839    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-model-cache\") pod \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") "
Oct 29 23:16:12.075892 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.075884    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-home\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-home\") pod \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") "
Oct 29 23:16:12.076059 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.075952    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tls-certs\" (UniqueName: \"kubernetes.io/secret/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tls-certs\") pod \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") "
Oct 29 23:16:12.076059 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.075994    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-dshm\") pod \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") "
Oct 29 23:16:12.076059 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.076033    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-kube-api-access-zsd9k\" (UniqueName: \"kubernetes.io/projected/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-kube-api-access-zsd9k\") pod \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") "
Oct 29 23:16:12.076059 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.076058    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tmp-dir\") pod \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\" (UID: \"18e76f05-d6fc-4d40-b0a9-58f976b8ea70\") "
Oct 29 23:16:12.076291 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.076114    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-model-cache" pod "18e76f05-d6fc-4d40-b0a9-58f976b8ea70" (UID: "18e76f05-d6fc-4d40-b0a9-58f976b8ea70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:16:12.076291 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.076209    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-home" pod "18e76f05-d6fc-4d40-b0a9-58f976b8ea70" (UID: "18e76f05-d6fc-4d40-b0a9-58f976b8ea70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:16:12.076432 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.076350    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:12.076432 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.076372    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:12.076432 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.076413    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tmp-dir" pod "18e76f05-d6fc-4d40-b0a9-58f976b8ea70" (UID: "18e76f05-d6fc-4d40-b0a9-58f976b8ea70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:16:12.078360 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.078319    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-dshm" pod "18e76f05-d6fc-4d40-b0a9-58f976b8ea70" (UID: "18e76f05-d6fc-4d40-b0a9-58f976b8ea70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:16:12.078445 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.078339    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-kube-api-access-zsd9k" pod "18e76f05-d6fc-4d40-b0a9-58f976b8ea70" (UID: "18e76f05-d6fc-4d40-b0a9-58f976b8ea70"). InnerVolumeSpecName "kube-api-access-zsd9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:16:12.078445 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.078403    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tls-certs" pod "18e76f05-d6fc-4d40-b0a9-58f976b8ea70" (UID: "18e76f05-d6fc-4d40-b0a9-58f976b8ea70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:16:12.177185 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.177157    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:12.177185 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.177182    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsd9k\" (UniqueName: \"kubernetes.io/projected/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-kube-api-access-zsd9k\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:12.177388 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.177194    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:12.177388 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:12.177203    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e76f05-d6fc-4d40-b0a9-58f976b8ea70-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:13.045491 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:13.045456    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"
Oct 29 23:16:13.096544 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:13.096513    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"]
Oct 29 23:16:13.108897 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:13.108868    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-1fe7678b-kserve-596d5b44b5-4pdqq"]
Oct 29 23:16:14.109624 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:14.109590    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e76f05-d6fc-4d40-b0a9-58f976b8ea70" path="/var/lib/kubelet/pods/18e76f05-d6fc-4d40-b0a9-58f976b8ea70/volumes"
Oct 29 23:16:16.161221 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.161184    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx" podUID="373a3fda-5bdf-49b6-b0e2-250d55c0d1f3" containerName="main" containerID="cri-o://52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6" gracePeriod=45
Oct 29 23:16:16.601631 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.601315    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:16:16.646916 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.646866    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-home\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-home\") pod \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") "
Oct 29 23:16:16.647211 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.647186    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-home" pod "373a3fda-5bdf-49b6-b0e2-250d55c0d1f3" (UID: "373a3fda-5bdf-49b6-b0e2-250d55c0d1f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:16:16.647303 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.647280    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-model-cache\") pod \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") "
Oct 29 23:16:16.647401 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.647352    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tmp-dir\") pod \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") "
Oct 29 23:16:16.647644 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.647618    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-model-cache" pod "373a3fda-5bdf-49b6-b0e2-250d55c0d1f3" (UID: "373a3fda-5bdf-49b6-b0e2-250d55c0d1f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:16:16.647644 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.647628    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tmp-dir" pod "373a3fda-5bdf-49b6-b0e2-250d55c0d1f3" (UID: "373a3fda-5bdf-49b6-b0e2-250d55c0d1f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:16:16.647797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.647705    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tls-certs\") pod \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") "
Oct 29 23:16:16.647797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.647756    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-dshm\") pod \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") "
Oct 29 23:16:16.647797 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.647786    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-kube-api-access-r2qlx\" (UniqueName: \"kubernetes.io/projected/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-kube-api-access-r2qlx\") pod \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\" (UID: \"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3\") "
Oct 29 23:16:16.648239 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.648219    4860 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-home\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:16.648407 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.648387    4860 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-model-cache\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:16.648489 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.648414    4860 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tmp-dir\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:16.649752 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.649731    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-dshm" pod "373a3fda-5bdf-49b6-b0e2-250d55c0d1f3" (UID: "373a3fda-5bdf-49b6-b0e2-250d55c0d1f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:16:16.650455 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.650436    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tls-certs" pod "373a3fda-5bdf-49b6-b0e2-250d55c0d1f3" (UID: "373a3fda-5bdf-49b6-b0e2-250d55c0d1f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:16:16.650602 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.650578    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-kube-api-access-r2qlx" pod "373a3fda-5bdf-49b6-b0e2-250d55c0d1f3" (UID: "373a3fda-5bdf-49b6-b0e2-250d55c0d1f3"). InnerVolumeSpecName "kube-api-access-r2qlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:16:16.750759 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.750651    4860 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-tls-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:16.750759 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.750689    4860 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-dshm\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:16.750759 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:16.750732    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2qlx\" (UniqueName: \"kubernetes.io/projected/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3-kube-api-access-r2qlx\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:17.045265 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.045220    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"]
Oct 29 23:16:17.049469 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.049444    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03fc0ef3-v2qf2"]
Oct 29 23:16:17.140352 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.128416    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:17.148599 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.140001    4860 generic.go:356] "Generic (PLEG): container finished" podID="373a3fda-5bdf-49b6-b0e2-250d55c0d1f3" containerID="52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6" exitCode=0
Oct 29 23:16:17.148877 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.140026    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx" event={"ID":"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3","Type":"ContainerDied","Data":"52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6"}
Oct 29 23:16:17.149031 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.149010    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx" event={"ID":"373a3fda-5bdf-49b6-b0e2-250d55c0d1f3","Type":"ContainerDied","Data":"945a8bcb4a7f413e8ddeae54bd6508b1d6e51ab84d97fb90961415e8fc01a67d"}
Oct 29 23:16:17.149369 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.140150    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"
Oct 29 23:16:17.149726 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.149219    4860 scope.go:122] "RemoveContainer" containerID="52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6"
Oct 29 23:16:17.163070 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.162866    4860 scope.go:122] "RemoveContainer" containerID="52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6"
Oct 29 23:16:17.163307 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:16:17.163142    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6\": container with ID starting with 52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6 not found: ID does not exist" containerID="52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6"
Oct 29 23:16:17.163307 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.163169    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6"} err="failed to get container status \"52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6\": rpc error: code = NotFound desc = could not find container \"52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6\": container with ID starting with 52b9cebae2958a9132ce8f26f817b55b59fe1bbe23dbf3b3778f20301d2812e6 not found: ID does not exist"
Oct 29 23:16:17.241001 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.240623    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"]
Oct 29 23:16:17.241001 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.240685    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-1fe7678b-kserve-55c595c969-vmgsx"]
Oct 29 23:16:17.479351 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:17.478365    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:18.059969 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:16:18.059930    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:16:18.113284 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:18.113247    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373a3fda-5bdf-49b6-b0e2-250d55c0d1f3" path="/var/lib/kubelet/pods/373a3fda-5bdf-49b6-b0e2-250d55c0d1f3/volumes"
Oct 29 23:16:18.113920 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:18.113898    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cdcf4b0-cbd3-4dab-9875-8955142c5a43" path="/var/lib/kubelet/pods/7cdcf4b0-cbd3-4dab-9875-8955142c5a43/volumes"
Oct 29 23:16:25.654220 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:25.654190    4860 scope.go:122] "RemoveContainer" containerID="24d16d4bb8e69c0f7406be63582f51ab2379de20dbc911e226e2f0711ccf841c"
Oct 29 23:16:25.663073 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:25.663056    4860 scope.go:122] "RemoveContainer" containerID="cd9f48cb735f50a23ee8aa3a761bc7f576ca3ad8504e11b2d6e9e11adc837a98"
Oct 29 23:16:25.670989 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:25.670970    4860 scope.go:122] "RemoveContainer" containerID="49773f3ecf9464e2023cecc530f904facf8fc3b2da6dc1d4fa0d3e378996cefe"
Oct 29 23:16:27.194670 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.194632    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"]
Oct 29 23:16:27.195093 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.195062    4860 cpu_manager.go:414] "containerMap: removing container" podUID="373a3fda-5bdf-49b6-b0e2-250d55c0d1f3" containerName="main"
Oct 29 23:16:27.195093 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.195075    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373a3fda-5bdf-49b6-b0e2-250d55c0d1f3" containerName="main"
Oct 29 23:16:27.195226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.195103    4860 cpu_manager.go:414] "containerMap: removing container" podUID="18e76f05-d6fc-4d40-b0a9-58f976b8ea70" containerName="main"
Oct 29 23:16:27.195226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.195112    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e76f05-d6fc-4d40-b0a9-58f976b8ea70" containerName="main"
Oct 29 23:16:27.195226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.195189    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e76f05-d6fc-4d40-b0a9-58f976b8ea70" containerName="main"
Oct 29 23:16:27.195226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.195206    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="373a3fda-5bdf-49b6-b0e2-250d55c0d1f3" containerName="main"
Oct 29 23:16:27.199757 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.199740    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:27.215379 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.215355    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"]
Oct 29 23:16:27.225052 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.225028    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:16:27.225052 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.225044    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:16:27.225052 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.225058    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:16:27.227089 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.227070    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vhd49\""
Oct 29 23:16:27.277688 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.277661    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7182ba-mk5lk\" (UID: \"bc511318-bcb4-43a4-9a2c-ec273445f320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:27.277802 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.277727    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7182ba-mk5lk\" (UID: \"bc511318-bcb4-43a4-9a2c-ec273445f320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:27.277855 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.277832    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqm9d\" (UniqueName: \"kubernetes.io/projected/bc511318-bcb4-43a4-9a2c-ec273445f320-kube-api-access-cqm9d\") pod \"maas-api-revoke-sub-9c7182ba-mk5lk\" (UID: \"bc511318-bcb4-43a4-9a2c-ec273445f320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:27.378805 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.378775    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqm9d\" (UniqueName: \"kubernetes.io/projected/bc511318-bcb4-43a4-9a2c-ec273445f320-kube-api-access-cqm9d\") pod \"maas-api-revoke-sub-9c7182ba-mk5lk\" (UID: \"bc511318-bcb4-43a4-9a2c-ec273445f320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:27.378979 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.378838    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7182ba-mk5lk\" (UID: \"bc511318-bcb4-43a4-9a2c-ec273445f320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:27.378979 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.378900    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7182ba-mk5lk\" (UID: \"bc511318-bcb4-43a4-9a2c-ec273445f320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:27.379654 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.379629    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7182ba-mk5lk\" (UID: \"bc511318-bcb4-43a4-9a2c-ec273445f320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:27.381198 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.381176    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7182ba-mk5lk\" (UID: \"bc511318-bcb4-43a4-9a2c-ec273445f320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:27.398550 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.398522    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqm9d\" (UniqueName: \"kubernetes.io/projected/bc511318-bcb4-43a4-9a2c-ec273445f320-kube-api-access-cqm9d\") pod \"maas-api-revoke-sub-9c7182ba-mk5lk\" (UID: \"bc511318-bcb4-43a4-9a2c-ec273445f320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:27.512466 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.512412    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:27.808040 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:16:27.808003    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc511318_bcb4_43a4_9a2c_ec273445f320.slice/crio-b831ca3c324745837177097f1e768be23ec6ebdd0a760efb2374cd935f1a3467 WatchSource:0}: Error finding container b831ca3c324745837177097f1e768be23ec6ebdd0a760efb2374cd935f1a3467: Status 404 returned error can't find the container with id b831ca3c324745837177097f1e768be23ec6ebdd0a760efb2374cd935f1a3467
Oct 29 23:16:27.808154 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:27.808110    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"]
Oct 29 23:16:28.259575 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:28.259537    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk" event={"ID":"bc511318-bcb4-43a4-9a2c-ec273445f320","Type":"ContainerStarted","Data":"d1354c704859cb95b00a4259b13f8714e42e0ec846c28ae39e94453af03ffb45"}
Oct 29 23:16:28.259945 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:28.259600    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk" event={"ID":"bc511318-bcb4-43a4-9a2c-ec273445f320","Type":"ContainerStarted","Data":"b831ca3c324745837177097f1e768be23ec6ebdd0a760efb2374cd935f1a3467"}
Oct 29 23:16:28.300175 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:28.300127    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk" podStartSLOduration=1.3001121310000001 podStartE2EDuration="1.300112131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:28.297661322 +0000 UTC m=+3730.322437474" watchObservedRunningTime="2026-10-29 23:16:28.300112131 +0000 UTC m=+3730.324888268"
Oct 29 23:16:29.267097 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:29.267064    4860 generic.go:356] "Generic (PLEG): container finished" podID="bc511318-bcb4-43a4-9a2c-ec273445f320" containerID="d1354c704859cb95b00a4259b13f8714e42e0ec846c28ae39e94453af03ffb45" exitCode=0
Oct 29 23:16:29.267543 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:29.267150    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk" event={"ID":"bc511318-bcb4-43a4-9a2c-ec273445f320","Type":"ContainerDied","Data":"d1354c704859cb95b00a4259b13f8714e42e0ec846c28ae39e94453af03ffb45"}
Oct 29 23:16:30.409483 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:30.409457    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:30.508238 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:30.508208    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-xks-ca\") pod \"bc511318-bcb4-43a4-9a2c-ec273445f320\" (UID: \"bc511318-bcb4-43a4-9a2c-ec273445f320\") "
Oct 29 23:16:30.508398 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:30.508299    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc511318-bcb4-43a4-9a2c-ec273445f320-kube-api-access-cqm9d\" (UniqueName: \"kubernetes.io/projected/bc511318-bcb4-43a4-9a2c-ec273445f320-kube-api-access-cqm9d\") pod \"bc511318-bcb4-43a4-9a2c-ec273445f320\" (UID: \"bc511318-bcb4-43a4-9a2c-ec273445f320\") "
Oct 29 23:16:30.508398 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:30.508368    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-openshift-ca\") pod \"bc511318-bcb4-43a4-9a2c-ec273445f320\" (UID: \"bc511318-bcb4-43a4-9a2c-ec273445f320\") "
Oct 29 23:16:30.508728 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:30.508705    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-openshift-ca" pod "bc511318-bcb4-43a4-9a2c-ec273445f320" (UID: "bc511318-bcb4-43a4-9a2c-ec273445f320"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:16:30.510484 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:30.510461    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc511318-bcb4-43a4-9a2c-ec273445f320-kube-api-access-cqm9d" pod "bc511318-bcb4-43a4-9a2c-ec273445f320" (UID: "bc511318-bcb4-43a4-9a2c-ec273445f320"). InnerVolumeSpecName "kube-api-access-cqm9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:16:30.510564 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:30.510463    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-xks-ca" pod "bc511318-bcb4-43a4-9a2c-ec273445f320" (UID: "bc511318-bcb4-43a4-9a2c-ec273445f320"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:16:30.609518 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:30.609464    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-xks-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:30.609518 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:30.609486    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqm9d\" (UniqueName: \"kubernetes.io/projected/bc511318-bcb4-43a4-9a2c-ec273445f320-kube-api-access-cqm9d\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:30.609518 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:30.609496    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc511318-bcb4-43a4-9a2c-ec273445f320-maas-api-openshift-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:31.278592 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:31.278566    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk"
Oct 29 23:16:31.278788 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:31.278562    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7182ba-mk5lk" event={"ID":"bc511318-bcb4-43a4-9a2c-ec273445f320","Type":"ContainerDied","Data":"b831ca3c324745837177097f1e768be23ec6ebdd0a760efb2374cd935f1a3467"}
Oct 29 23:16:31.278788 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:31.278685    4860 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b831ca3c324745837177097f1e768be23ec6ebdd0a760efb2374cd935f1a3467"
Oct 29 23:16:37.779735 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:37.779699    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:39.643810 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.643775    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"]
Oct 29 23:16:39.644238 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.644197    4860 cpu_manager.go:414] "containerMap: removing container" podUID="bc511318-bcb4-43a4-9a2c-ec273445f320" containerName="revoke-keys"
Oct 29 23:16:39.644238 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.644211    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc511318-bcb4-43a4-9a2c-ec273445f320" containerName="revoke-keys"
Oct 29 23:16:39.644364 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.644305    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc511318-bcb4-43a4-9a2c-ec273445f320" containerName="revoke-keys"
Oct 29 23:16:39.647577 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.647552    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.665843 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.665811    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"]
Oct 29 23:16:39.670201 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.670179    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.671729 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.671671    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"]
Oct 29 23:16:39.673702 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.673682    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 29 23:16:39.673987 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.673962    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dzwhk\""
Oct 29 23:16:39.681135 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.681093    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 23:16:39.681393 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.681356    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:16:39.689377 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.686291    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:16:39.689377 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.689307    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"]
Oct 29 23:16:39.791081 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.791048    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393923b1-5e09-47f9-87a9-f66cc04051d2-openshift-service-ca\") pod \"maas-api-5c48877c75-2qkps\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.791231 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.791089    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f747250-b165-4395-a926-0220093db525-metrics-certs\") pod \"maas-api-595cc648c-zp2cg\" (UID: \"1f747250-b165-4395-a926-0220093db525\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.791231 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.791167    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f747250-b165-4395-a926-0220093db525-maas-api-tls\") pod \"maas-api-595cc648c-zp2cg\" (UID: \"1f747250-b165-4395-a926-0220093db525\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.791231 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.791187    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-maas-api-tls\") pod \"maas-api-5c48877c75-2qkps\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.791231 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.791207    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwzlk\" (UniqueName: \"kubernetes.io/projected/393923b1-5e09-47f9-87a9-f66cc04051d2-kube-api-access-fwzlk\") pod \"maas-api-5c48877c75-2qkps\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.791411 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.791292    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f747250-b165-4395-a926-0220093db525-openshift-service-ca\") pod \"maas-api-595cc648c-zp2cg\" (UID: \"1f747250-b165-4395-a926-0220093db525\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.791411 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.791309    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brdx\" (UniqueName: \"kubernetes.io/projected/1f747250-b165-4395-a926-0220093db525-kube-api-access-9brdx\") pod \"maas-api-595cc648c-zp2cg\" (UID: \"1f747250-b165-4395-a926-0220093db525\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.791480 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.791451    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-metrics-certs\") pod \"maas-api-5c48877c75-2qkps\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.892192 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.892166    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f747250-b165-4395-a926-0220093db525-maas-api-tls\") pod \"maas-api-595cc648c-zp2cg\" (UID: \"1f747250-b165-4395-a926-0220093db525\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.892192 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.892194    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-maas-api-tls\") pod \"maas-api-5c48877c75-2qkps\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.892417 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.892210    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwzlk\" (UniqueName: \"kubernetes.io/projected/393923b1-5e09-47f9-87a9-f66cc04051d2-kube-api-access-fwzlk\") pod \"maas-api-5c48877c75-2qkps\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.892417 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.892234    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f747250-b165-4395-a926-0220093db525-openshift-service-ca\") pod \"maas-api-595cc648c-zp2cg\" (UID: \"1f747250-b165-4395-a926-0220093db525\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.892417 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.892257    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brdx\" (UniqueName: \"kubernetes.io/projected/1f747250-b165-4395-a926-0220093db525-kube-api-access-9brdx\") pod \"maas-api-595cc648c-zp2cg\" (UID: \"1f747250-b165-4395-a926-0220093db525\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.892417 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.892352    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-metrics-certs\") pod \"maas-api-5c48877c75-2qkps\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.892417 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.892387    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393923b1-5e09-47f9-87a9-f66cc04051d2-openshift-service-ca\") pod \"maas-api-5c48877c75-2qkps\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.892702 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.892425    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f747250-b165-4395-a926-0220093db525-metrics-certs\") pod \"maas-api-595cc648c-zp2cg\" (UID: \"1f747250-b165-4395-a926-0220093db525\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.893120 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.893087    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f747250-b165-4395-a926-0220093db525-openshift-service-ca\") pod \"maas-api-595cc648c-zp2cg\" (UID: \"1f747250-b165-4395-a926-0220093db525\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.893226 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.893202    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393923b1-5e09-47f9-87a9-f66cc04051d2-openshift-service-ca\") pod \"maas-api-5c48877c75-2qkps\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.894910 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.894845    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-metrics-certs\") pod \"maas-api-5c48877c75-2qkps\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.895000 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.894908    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f747250-b165-4395-a926-0220093db525-metrics-certs\") pod \"maas-api-595cc648c-zp2cg\" (UID: \"1f747250-b165-4395-a926-0220093db525\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.895000 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.894906    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-maas-api-tls\") pod \"maas-api-5c48877c75-2qkps\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.895082 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.895014    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f747250-b165-4395-a926-0220093db525-maas-api-tls\") pod \"maas-api-595cc648c-zp2cg\" (UID: \"1f747250-b165-4395-a926-0220093db525\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.913045 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.913022    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwzlk\" (UniqueName: \"kubernetes.io/projected/393923b1-5e09-47f9-87a9-f66cc04051d2-kube-api-access-fwzlk\") pod \"maas-api-5c48877c75-2qkps\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.913597 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.913574    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brdx\" (UniqueName: \"kubernetes.io/projected/1f747250-b165-4395-a926-0220093db525-kube-api-access-9brdx\") pod \"maas-api-595cc648c-zp2cg\" (UID: \"1f747250-b165-4395-a926-0220093db525\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:39.962097 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.962073    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:39.985260 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:39.985221    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:40.265472 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:16:40.265443    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393923b1_5e09_47f9_87a9_f66cc04051d2.slice/crio-f0385d23812bda3bb440fedf45586a77a2146fd2d204efbc9a6d3d6fbc8f796c WatchSource:0}: Error finding container f0385d23812bda3bb440fedf45586a77a2146fd2d204efbc9a6d3d6fbc8f796c: Status 404 returned error can't find the container with id f0385d23812bda3bb440fedf45586a77a2146fd2d204efbc9a6d3d6fbc8f796c
Oct 29 23:16:40.266042 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:40.266017    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"]
Oct 29 23:16:40.266755 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:40.266736    4860 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:16:40.302843 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:16:40.302813    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f747250_b165_4395_a926_0220093db525.slice/crio-ee382d63c1c5672ee3e14a31fc0994721ddb6aeb518a754502ab4dc435734796 WatchSource:0}: Error finding container ee382d63c1c5672ee3e14a31fc0994721ddb6aeb518a754502ab4dc435734796: Status 404 returned error can't find the container with id ee382d63c1c5672ee3e14a31fc0994721ddb6aeb518a754502ab4dc435734796
Oct 29 23:16:40.303851 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:40.303822    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"]
Oct 29 23:16:40.328476 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:40.328450    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg" event={"ID":"1f747250-b165-4395-a926-0220093db525","Type":"ContainerStarted","Data":"ee382d63c1c5672ee3e14a31fc0994721ddb6aeb518a754502ab4dc435734796"}
Oct 29 23:16:40.329689 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:40.329668    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps" event={"ID":"393923b1-5e09-47f9-87a9-f66cc04051d2","Type":"ContainerStarted","Data":"f0385d23812bda3bb440fedf45586a77a2146fd2d204efbc9a6d3d6fbc8f796c"}
Oct 29 23:16:41.337205 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:41.337165    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps" event={"ID":"393923b1-5e09-47f9-87a9-f66cc04051d2","Type":"ContainerStarted","Data":"1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a"}
Oct 29 23:16:41.337719 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:41.337238    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:41.339027 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:41.338999    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg" event={"ID":"1f747250-b165-4395-a926-0220093db525","Type":"ContainerStarted","Data":"1b09356fb7bf3e94563c01232755fef853edabe27f850538bf85d0caddb895f3"}
Oct 29 23:16:41.339165 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:41.339078    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:41.378610 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:41.378564    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps" podStartSLOduration=1.9770704289999999 podStartE2EDuration="2.37855196s" totalImagesPullingTime="401.481531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:41.377062001 +0000 UTC m=+3743.401838158" watchObservedRunningTime="2026-10-29 23:16:41.37855196 +0000 UTC m=+3743.403328136"
Oct 29 23:16:41.417217 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:41.417172    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg" podStartSLOduration=2.089072416 podStartE2EDuration="2.417159542s" totalImagesPullingTime="328.087126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:41.416594267 +0000 UTC m=+3743.441370411" watchObservedRunningTime="2026-10-29 23:16:41.417159542 +0000 UTC m=+3743.441935672"
Oct 29 23:16:47.350003 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.349971    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:47.350499 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.350429    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-zp2cg"
Oct 29 23:16:47.463766 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.463729    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"]
Oct 29 23:16:47.463997 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.463948    4860 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps" podUID="393923b1-5e09-47f9-87a9-f66cc04051d2" containerName="maas-api" containerID="cri-o://1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a" gracePeriod=30
Oct 29 23:16:47.720830 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.720806    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:47.860613 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.860579    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-metrics-certs\") pod \"393923b1-5e09-47f9-87a9-f66cc04051d2\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") "
Oct 29 23:16:47.860778 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.860617    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-maas-api-tls\") pod \"393923b1-5e09-47f9-87a9-f66cc04051d2\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") "
Oct 29 23:16:47.860778 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.860664    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393923b1-5e09-47f9-87a9-f66cc04051d2-kube-api-access-fwzlk\" (UniqueName: \"kubernetes.io/projected/393923b1-5e09-47f9-87a9-f66cc04051d2-kube-api-access-fwzlk\") pod \"393923b1-5e09-47f9-87a9-f66cc04051d2\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") "
Oct 29 23:16:47.860778 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.860732    4860 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393923b1-5e09-47f9-87a9-f66cc04051d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393923b1-5e09-47f9-87a9-f66cc04051d2-openshift-service-ca\") pod \"393923b1-5e09-47f9-87a9-f66cc04051d2\" (UID: \"393923b1-5e09-47f9-87a9-f66cc04051d2\") "
Oct 29 23:16:47.861185 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.861158    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393923b1-5e09-47f9-87a9-f66cc04051d2-openshift-service-ca" pod "393923b1-5e09-47f9-87a9-f66cc04051d2" (UID: "393923b1-5e09-47f9-87a9-f66cc04051d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:16:47.862721 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.862690    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-metrics-certs" pod "393923b1-5e09-47f9-87a9-f66cc04051d2" (UID: "393923b1-5e09-47f9-87a9-f66cc04051d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:16:47.862721 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.862708    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-maas-api-tls" pod "393923b1-5e09-47f9-87a9-f66cc04051d2" (UID: "393923b1-5e09-47f9-87a9-f66cc04051d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:16:47.862868 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.862804    4860 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393923b1-5e09-47f9-87a9-f66cc04051d2-kube-api-access-fwzlk" pod "393923b1-5e09-47f9-87a9-f66cc04051d2" (UID: "393923b1-5e09-47f9-87a9-f66cc04051d2"). InnerVolumeSpecName "kube-api-access-fwzlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:16:47.961471 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.961446    4860 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-metrics-certs\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:47.961471 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.961468    4860 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393923b1-5e09-47f9-87a9-f66cc04051d2-maas-api-tls\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:47.961625 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.961477    4860 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwzlk\" (UniqueName: \"kubernetes.io/projected/393923b1-5e09-47f9-87a9-f66cc04051d2-kube-api-access-fwzlk\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:47.961625 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:47.961489    4860 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393923b1-5e09-47f9-87a9-f66cc04051d2-openshift-service-ca\") on node \"ip-10-0-128-31.ec2.internal\" DevicePath \"\""
Oct 29 23:16:48.053651 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:16:48.053614    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:16:48.379742 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:48.379656    4860 generic.go:356] "Generic (PLEG): container finished" podID="393923b1-5e09-47f9-87a9-f66cc04051d2" containerID="1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a" exitCode=0
Oct 29 23:16:48.379742 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:48.379697    4860 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"
Oct 29 23:16:48.380274 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:48.379737    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps" event={"ID":"393923b1-5e09-47f9-87a9-f66cc04051d2","Type":"ContainerDied","Data":"1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a"}
Oct 29 23:16:48.380274 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:48.379797    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2qkps" event={"ID":"393923b1-5e09-47f9-87a9-f66cc04051d2","Type":"ContainerDied","Data":"f0385d23812bda3bb440fedf45586a77a2146fd2d204efbc9a6d3d6fbc8f796c"}
Oct 29 23:16:48.380274 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:48.379829    4860 scope.go:122] "RemoveContainer" containerID="1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a"
Oct 29 23:16:48.389152 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:48.389135    4860 scope.go:122] "RemoveContainer" containerID="1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a"
Oct 29 23:16:48.389421 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:16:48.389404    4860 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a\": container with ID starting with 1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a not found: ID does not exist" containerID="1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a"
Oct 29 23:16:48.389478 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:48.389430    4860 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a"} err="failed to get container status \"1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a\": rpc error: code = NotFound desc = could not find container \"1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a\": container with ID starting with 1bc4a438a053ad1c1a64cbf63698976f5ae290dd265d0c474f5d9575b7e3a46a not found: ID does not exist"
Oct 29 23:16:48.432844 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:48.432821    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"]
Oct 29 23:16:48.445901 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:48.445879    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-2qkps"]
Oct 29 23:16:50.135363 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:16:50.130206    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393923b1-5e09-47f9-87a9-f66cc04051d2" path="/var/lib/kubelet/pods/393923b1-5e09-47f9-87a9-f66cc04051d2/volumes"
Oct 29 23:17:14.441102 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:14.440463    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:17:14.441102 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:14.440550    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vkx5n" podUID="cc9cb970-f2d1-4e47-be8a-20e20982ff32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:17:14.466444 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:14.466418    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzv89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:17:14.466584 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:14.466473    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzv89" podUID="e280bf1d-cfc4-48ba-8531-720a62ebd898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:17:14.738820 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:14.738418    4860 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-96ppp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:17:14.738820 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:14.738478    4860 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-96ppp" podUID="58904265-85eb-43c4-9f23-dbaa1a682acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:17:17.214315 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:17.214279    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:18.057070 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:17:18.057029    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:17:25.705714 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:25.705684    4860 scope.go:122] "RemoveContainer" containerID="1098b32770d4212c284a7f80efee806a1995496b183e66434bdc3eabdf25f488"
Oct 29 23:17:25.714645 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:25.714628    4860 scope.go:122] "RemoveContainer" containerID="87b5f24286f66f9d63f45fbebd1410d997f6d18f79107bcfcdf03d4e19dd225e"
Oct 29 23:17:37.678870 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:37.678836    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c78df8ff-z7jjx_876e4ae8-d438-444e-94fa-d603285f7fd8/authorino/0.log"
Oct 29 23:17:48.011550 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:48.011509    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5955b644bc-t5bnn_a5df48c6-245a-4387-8d8f-745e0ef5d0f1/manager/0.log"
Oct 29 23:17:48.054375 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:17:48.054289    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:17:50.104893 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:50.104860    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f7c578db4-f4vnd_6e67bb2d-9995-49bb-ba2c-00950cc17aaa/copy-manifests/0.log"
Oct 29 23:17:50.143689 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:50.143661    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f7c578db4-f4vnd_6e67bb2d-9995-49bb-ba2c-00950cc17aaa/manager/0.log"
Oct 29 23:17:51.523390 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:51.523362    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-zp2cg_1f747250-b165-4395-a926-0220093db525/maas-api/0.log"
Oct 29 23:17:52.429524 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:52.429498    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9c7182ba-mk5lk_bc511318-bcb4-43a4-9a2c-ec273445f320/revoke-keys/0.log"
Oct 29 23:17:52.602547 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:52.602520    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-aa147a01-dnv7m_1693e04a-5d59-4749-8b3d-8d26373d9e1e/revoke-keys/0.log"
Oct 29 23:17:52.797170 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:52.797148    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-6hg6t_963dfe0e-ca6f-4421-88a3-6a92352c17d4/revoke-keys/0.log"
Oct 29 23:17:52.968276 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:52.968251    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-7cmrf_2810b57f-2f9a-4e4b-bc6e-0291c3f203de/revoke-keys/0.log"
Oct 29 23:17:54.851024 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:54.850977    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c78df8ff-z7jjx_876e4ae8-d438-444e-94fa-d603285f7fd8/authorino/0.log"
Oct 29 23:17:55.456957 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:55.456929    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-v9gbd_c70a71db-8aee-4f26-b5ef-abbcfcaf4440/kuadrant-console-plugin/0.log"
Oct 29 23:17:57.440578 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:57.440550    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b84b87b7c-vx6bc_59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e/kube-auth-proxy/0.log"
Oct 29 23:17:57.807024 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:57.806946    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-96ppp_58904265-85eb-43c4-9f23-dbaa1a682acc/istio-proxy/0.log"
Oct 29 23:17:59.097212 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:59.097184    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-vkx5n_cc9cb970-f2d1-4e47-be8a-20e20982ff32/istio-proxy/0.log"
Oct 29 23:17:59.290981 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:59.290938    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zzv89_e280bf1d-cfc4-48ba-8531-720a62ebd898/istio-proxy/0.log"
Oct 29 23:17:59.476360 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:59.476311    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-64t9f_ac5ce19f-bfe6-41bd-9c75-70dd54594ea3/payload-pre-processing/0.log"
Oct 29 23:17:59.640738 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:59.640683    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-8wn8h_23a26fe4-1010-464f-9df0-ff85860cc488/payload-processing/0.log"
Oct 29 23:17:59.820652 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:17:59.820595    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b6b64f55f-9l4xp_19e92d72-5f01-4c0f-9739-e094b74d5825/router/0.log"
Oct 29 23:18:00.803453 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:00.803424    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs_faba6185-f761-4f97-b100-653f1d78ca35/main/0.log"
Oct 29 23:18:01.184986 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:01.184951    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2p75h_f34300ae-1300-4d57-af82-cb727c236caf/main/0.log"
Oct 29 23:18:01.556535 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:01.556462    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb_82c8ba42-03e9-4192-a9af-6bdb5ceec1a5/main/0.log"
Oct 29 23:18:01.949481 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:01.949453    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m_da28494b-2ff0-4782-b53f-2583dc161b9d/main/0.log"
Oct 29 23:18:02.819065 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:02.819036    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c78df8ff-z7jjx_876e4ae8-d438-444e-94fa-d603285f7fd8/authorino/0.log"
Oct 29 23:18:11.666204 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:11.666170    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5955b644bc-t5bnn_a5df48c6-245a-4387-8d8f-745e0ef5d0f1/manager/0.log"
Oct 29 23:18:13.568441 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:13.568408    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f7c578db4-f4vnd_6e67bb2d-9995-49bb-ba2c-00950cc17aaa/manager/0.log"
Oct 29 23:18:13.595879 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:13.595856    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f7c578db4-f4vnd_6e67bb2d-9995-49bb-ba2c-00950cc17aaa/copy-manifests/0.log"
Oct 29 23:18:14.732249 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:14.732217    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-zp2cg_1f747250-b165-4395-a926-0220093db525/maas-api/0.log"
Oct 29 23:18:15.626241 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:15.626213    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9c7182ba-mk5lk_bc511318-bcb4-43a4-9a2c-ec273445f320/revoke-keys/0.log"
Oct 29 23:18:15.804304 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:15.804277    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-aa147a01-dnv7m_1693e04a-5d59-4749-8b3d-8d26373d9e1e/revoke-keys/0.log"
Oct 29 23:18:15.979579 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:15.979550    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-6hg6t_963dfe0e-ca6f-4421-88a3-6a92352c17d4/revoke-keys/0.log"
Oct 29 23:18:16.137155 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:16.137127    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-7cmrf_2810b57f-2f9a-4e4b-bc6e-0291c3f203de/revoke-keys/0.log"
Oct 29 23:18:17.433783 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:17.433747    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:17.623659 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:17.623631    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c78df8ff-z7jjx_876e4ae8-d438-444e-94fa-d603285f7fd8/authorino/0.log"
Oct 29 23:18:18.054518 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:18:18.054485    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:18:18.226794 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:18.226761    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-v9gbd_c70a71db-8aee-4f26-b5ef-abbcfcaf4440/kuadrant-console-plugin/0.log"
Oct 29 23:18:19.957023 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:19.956993    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b84b87b7c-vx6bc_59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e/kube-auth-proxy/0.log"
Oct 29 23:18:20.365578 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:20.365491    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-96ppp_58904265-85eb-43c4-9f23-dbaa1a682acc/istio-proxy/0.log"
Oct 29 23:18:21.738722 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:21.738692    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-vkx5n_cc9cb970-f2d1-4e47-be8a-20e20982ff32/istio-proxy/0.log"
Oct 29 23:18:21.917592 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:21.917563    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zzv89_e280bf1d-cfc4-48ba-8531-720a62ebd898/istio-proxy/0.log"
Oct 29 23:18:22.100151 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:22.100062    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-64t9f_ac5ce19f-bfe6-41bd-9c75-70dd54594ea3/payload-pre-processing/0.log"
Oct 29 23:18:22.278941 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:22.278915    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-8wn8h_23a26fe4-1010-464f-9df0-ff85860cc488/payload-processing/0.log"
Oct 29 23:18:22.461494 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:22.461469    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b6b64f55f-9l4xp_19e92d72-5f01-4c0f-9739-e094b74d5825/router/0.log"
Oct 29 23:18:22.990609 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:22.990579    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-7zczs_faba6185-f761-4f97-b100-653f1d78ca35/main/0.log"
Oct 29 23:18:23.375763 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:23.375687    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2p75h_f34300ae-1300-4d57-af82-cb727c236caf/main/0.log"
Oct 29 23:18:23.778375 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:23.778348    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8654tb_82c8ba42-03e9-4192-a9af-6bdb5ceec1a5/main/0.log"
Oct 29 23:18:23.827025 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:23.827001    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:24.162298 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:24.162223    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fbn6m_da28494b-2ff0-4782-b53f-2583dc161b9d/main/0.log"
Oct 29 23:18:33.030339 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:33.030295    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"]
Oct 29 23:18:33.044876 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:33.044841    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7cmrf"]
Oct 29 23:18:34.110036 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:34.109995    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2810b57f-2f9a-4e4b-bc6e-0291c3f203de" path="/var/lib/kubelet/pods/2810b57f-2f9a-4e4b-bc6e-0291c3f203de/volumes"
Oct 29 23:18:48.028059 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:48.028025    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"]
Oct 29 23:18:48.039978 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:48.039946    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6hg6t"]
Oct 29 23:18:48.054312 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:18:48.054284    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:18:48.109452 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:48.109425    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963dfe0e-ca6f-4421-88a3-6a92352c17d4" path="/var/lib/kubelet/pods/963dfe0e-ca6f-4421-88a3-6a92352c17d4/volumes"
Oct 29 23:18:57.683003 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:57.682974    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-8wn8h_23a26fe4-1010-464f-9df0-ff85860cc488/payload-processing/0.log"
Oct 29 23:18:57.871842 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:18:57.871816    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-8wn8h_23a26fe4-1010-464f-9df0-ff85860cc488/payload-processing/0.log"
Oct 29 23:19:17.450153 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:17.450106    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:18.054224 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:19:18.054190    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:19:18.944044 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:18.944011    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 23:19:18.959352 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:18.959316    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 23:19:37.027975 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:37.027901    4860 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"]
Oct 29 23:19:37.040767 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:37.040741    4860 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa147a01-dnv7m"]
Oct 29 23:19:38.110036 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:38.109993    4860 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1693e04a-5d59-4749-8b3d-8d26373d9e1e" path="/var/lib/kubelet/pods/1693e04a-5d59-4749-8b3d-8d26373d9e1e/volumes"
Oct 29 23:19:40.144970 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:40.144939    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-8wn8h_23a26fe4-1010-464f-9df0-ff85860cc488/payload-processing/0.log"
Oct 29 23:19:40.312596 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:40.312567    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-8wn8h_23a26fe4-1010-464f-9df0-ff85860cc488/payload-processing/0.log"
Oct 29 23:19:40.645246 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:40.645213    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:42.043632 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:42.043597    4860 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log" limit=1024
Oct 29 23:19:45.426676 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:45.426652    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-stxld_7d56328b-9fa0-45cb-996f-65c1c1b8785c/global-pull-secret-syncer/0.log"
Oct 29 23:19:45.583595 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:45.583571    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-2z8f7_ccb78139-7ccc-4ac8-87bf-264d048488c4/connection-checker/0.log"
Oct 29 23:19:45.698032 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:45.698006    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6zmj5_1e6952c3-e47b-4c03-8804-04de63d8f1ee/connection-checker/0.log"
Oct 29 23:19:45.812346 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:45.812303    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-stxg2_31bc5aff-808a-4d45-bcac-80d1bd626d47/connection-checker/0.log"
Oct 29 23:19:45.945622 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:45.945579    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-99lmd_b6c8fff1-c08b-4437-944f-9ee33d186c42/konnectivity-agent/0.log"
Oct 29 23:19:46.358058 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:46.358018    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-31.ec2.internal_ed322e80a89b2bbeafbf0f88fe9597d6/haproxy/0.log"
Oct 29 23:19:48.054749 ip-10-0-128-31 kubenswrapper[4860]: E1029 23:19:48.054709    4860 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd9qn" containerName="extractor"
Oct 29 23:19:52.818476 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:52.818436    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c78df8ff-z7jjx_876e4ae8-d438-444e-94fa-d603285f7fd8/authorino/0.log"
Oct 29 23:19:53.180335 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:53.180303    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-v9gbd_c70a71db-8aee-4f26-b5ef-abbcfcaf4440/kuadrant-console-plugin/0.log"
Oct 29 23:19:55.020772 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.020729    4860 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"]
Oct 29 23:19:55.021164 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.021148    4860 cpu_manager.go:414] "containerMap: removing container" podUID="393923b1-5e09-47f9-87a9-f66cc04051d2" containerName="maas-api"
Oct 29 23:19:55.021164 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.021162    4860 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393923b1-5e09-47f9-87a9-f66cc04051d2" containerName="maas-api"
Oct 29 23:19:55.021261 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.021250    4860 memory_manager.go:365] "RemoveStaleState removing state" podUID="393923b1-5e09-47f9-87a9-f66cc04051d2" containerName="maas-api"
Oct 29 23:19:55.025559 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.025539    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.045423 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.045391    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"]
Oct 29 23:19:55.082451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.082424    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-podres\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.082587 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.082455    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-lib-modules\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.082587 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.082547    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-sys\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.082670 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.082626    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-proc\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.082711 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.082682    4860 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7d4\" (UniqueName: \"kubernetes.io/projected/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-kube-api-access-9c7d4\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.133663 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.133638    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mglbp\"/\"kube-root-ca.crt\""
Oct 29 23:19:55.133663 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.133638    4860 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mglbp\"/\"openshift-service-ca.crt\""
Oct 29 23:19:55.135166 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.135154    4860 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mglbp\"/\"default-dockercfg-d8vtb\""
Oct 29 23:19:55.183682 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.183657    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-sys\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.183831 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.183732    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-proc\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.183831 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.183762    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-proc\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.183831 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.183731    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-sys\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.183831 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.183789    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7d4\" (UniqueName: \"kubernetes.io/projected/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-kube-api-access-9c7d4\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.184071 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.183847    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-podres\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.184071 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.183886    4860 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-lib-modules\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.184071 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.183911    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-podres\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.184071 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.184000    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-lib-modules\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.203580 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.203557    4860 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7d4\" (UniqueName: \"kubernetes.io/projected/94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6-kube-api-access-9c7d4\") pod \"perf-node-gather-daemonset-jnhqv\" (UID: \"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6\") " pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.338788 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.338727    4860 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:55.633198 ip-10-0-128-31 kubenswrapper[4860]: W1029 23:19:55.633162    4860 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod94cfce72_f8d2_424e_aa1b_5dfbd7b7c9a6.slice/crio-009ad70be469cdb6fe2e094cdbaf9b038646f3991554f930bf9f05407e10f660 WatchSource:0}: Error finding container 009ad70be469cdb6fe2e094cdbaf9b038646f3991554f930bf9f05407e10f660: Status 404 returned error can't find the container with id 009ad70be469cdb6fe2e094cdbaf9b038646f3991554f930bf9f05407e10f660
Oct 29 23:19:55.633548 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.633529    4860 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"]
Oct 29 23:19:55.773738 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:55.773707    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-n98ww_f1afad65-7b45-46ac-b42d-d4cf7a5b8e8d/cluster-monitoring-operator/0.log"
Oct 29 23:19:56.456670 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:56.456644    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8stqn_8fd7ac88-64eb-4ae3-a3bc-587cf3b365be/node-exporter/0.log"
Oct 29 23:19:56.490185 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:56.490157    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv" event={"ID":"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6","Type":"ContainerStarted","Data":"08f6377720d25565fc06ec87ea40a789fd9a6a2854d7c6bf6264f965f4f64975"}
Oct 29 23:19:56.490350 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:56.490205    4860 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv" event={"ID":"94cfce72-f8d2-424e-aa1b-5dfbd7b7c9a6","Type":"ContainerStarted","Data":"009ad70be469cdb6fe2e094cdbaf9b038646f3991554f930bf9f05407e10f660"}
Oct 29 23:19:56.490350 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:56.490288    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:19:56.531299 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:56.531261    4860 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv" podStartSLOduration=2.5312503729999998 podStartE2EDuration="2.531250373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:56.529633651 +0000 UTC m=+3938.554409785" watchObservedRunningTime="2026-10-29 23:19:56.531250373 +0000 UTC m=+3938.556026503"
Oct 29 23:19:56.567850 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:56.567809    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8stqn_8fd7ac88-64eb-4ae3-a3bc-587cf3b365be/kube-rbac-proxy/0.log"
Oct 29 23:19:56.679253 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:19:56.679233    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8stqn_8fd7ac88-64eb-4ae3-a3bc-587cf3b365be/init-textfile/0.log"
Oct 29 23:20:01.447451 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:01.447421    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-w9grl_20f39049-d7a1-46d8-8ebc-c0159906d689/networking-console-plugin/0.log"
Oct 29 23:20:02.505257 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:02.505230    4860 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mglbp/perf-node-gather-daemonset-jnhqv"
Oct 29 23:20:02.525021 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:02.524990    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/2.log"
Oct 29 23:20:02.529544 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:02.529522    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7gx9p_7496f437-cb3c-4867-af37-16835f1134a1/console-operator/3.log"
Oct 29 23:20:04.861131 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:04.861108    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-vmtz2_6f6f3621-6d10-4afb-9438-c73cc3d4b13d/volume-data-source-validator/0.log"
Oct 29 23:20:07.434814 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:07.434788    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5sn5b_7099dc88-0238-49a5-b0fb-8d008e1b040e/dns/0.log"
Oct 29 23:20:07.544845 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:07.544815    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5sn5b_7099dc88-0238-49a5-b0fb-8d008e1b040e/kube-rbac-proxy/0.log"
Oct 29 23:20:08.134812 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:08.134777    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m7gqm_a4c49406-6e72-4bdd-8375-20dffea48c35/dns-node-resolver/0.log"
Oct 29 23:20:09.580051 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:09.580024    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x8qdl_93d782aa-ad96-4f81-a5a1-d356d022106b/node-ca/0.log"
Oct 29 23:20:10.516988 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:10.516958    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b84b87b7c-vx6bc_59ddbdc4-f0f7-4d14-ac0b-83dca35e4e8e/kube-auth-proxy/0.log"
Oct 29 23:20:10.636887 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:10.636860    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-96ppp_58904265-85eb-43c4-9f23-dbaa1a682acc/istio-proxy/0.log"
Oct 29 23:20:11.045757 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:11.045728    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-vkx5n_cc9cb970-f2d1-4e47-be8a-20e20982ff32/istio-proxy/0.log"
Oct 29 23:20:11.103641 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:11.103614    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zzv89_e280bf1d-cfc4-48ba-8531-720a62ebd898/istio-proxy/0.log"
Oct 29 23:20:11.161103 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:11.161072    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-64t9f_ac5ce19f-bfe6-41bd-9c75-70dd54594ea3/payload-pre-processing/0.log"
Oct 29 23:20:11.217185 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:11.217165    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-8wn8h_23a26fe4-1010-464f-9df0-ff85860cc488/payload-processing/0.log"
Oct 29 23:20:11.280157 ip-10-0-128-31 kubenswrapper[4860]: I1029 23:20:11.280132    4860 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7b6b64f55f-9l4xp_19e92d72-5f01-4c0f-9739-e094b74d5825/router/0.log"